2024-01-15T22:14:04.862 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-15T22:14:04.863 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --name yuriw-2024-01-15_22:02:29-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi --verbose --description rados/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/basic 3-final} --archive /home/teuthworker/archive/yuriw-2024-01-15_22:02:29-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7517238 -- /home/teuthworker/archive/yuriw-2024-01-15_22:02:29-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7517238/orig.config.yaml 2024-01-15T22:14:04.894 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-15T22:14:05.018 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-01-15_22:02:29-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7517238 branch: wip-yuri2-testing-2024-01-12-1128-pacific description: rados/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/basic 3-final} email: null first_in_suite: false job_id: '7517238' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-01-15_22:02:29-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-yuri2-testing-2024-01-12-1128-pacific ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 445562ab4bc3ddfb386936119050695810860bcb ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 445562ab4bc3ddfb386936119050695810860bcb selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri2-testing-2024-01-12-1128-pacific sha1: 445562ab4bc3ddfb386936119050695810860bcb owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - - host.b - client.1 seed: 5246 sha1: 445562ab4bc3ddfb386936119050695810860bcb sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri2-testing-2024-01-12-1128-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 445562ab4bc3ddfb386936119050695810860bcb targets: smithi099.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMot5Qaeq01nq2YXbQR2TDuQS4ijbfr9CnnRbMpJBClwb7TSaR9s+biN/qOmzGjqSKwc0rJptpRmHgw32mRGdUE= smithi162.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE9yp1RriOwRgcLJDD6X2eivuvf40mVddiFtyoo0tvH3VNXb4FgLO/0b5OsnIpZcsKUUvRfKEaq6ONa3eaRZJIc= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:3.0 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - nvme_loop: null - cephadm: roleless: true - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - stat -c '%u %g' /var/log/ceph | grep '167 167' - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch ls | grep '^osd.all-available-devices ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-15_22:02:29 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-01-15T22:14:05.019 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa; will attempt to use it 2024-01-15T22:14:05.020 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks 2024-01-15T22:14:05.020 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-15T22:14:05.021 INFO:teuthology.task.internal:Checking packages... 2024-01-15T22:14:05.049 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '445562ab4bc3ddfb386936119050695810860bcb' 2024-01-15T22:14:05.049 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-15T22:14:05.049 INFO:teuthology.packaging:ref: None 2024-01-15T22:14:05.049 INFO:teuthology.packaging:tag: None 2024-01-15T22:14:05.049 INFO:teuthology.packaging:branch: wip-yuri2-testing-2024-01-12-1128-pacific 2024-01-15T22:14:05.049 INFO:teuthology.packaging:sha1: 445562ab4bc3ddfb386936119050695810860bcb 2024-01-15T22:14:05.049 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri2-testing-2024-01-12-1128-pacific 2024-01-15T22:14:05.188 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-456.g445562ab 2024-01-15T22:14:05.189 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-15T22:14:05.196 INFO:teuthology.task.internal:no buildpackages task found 2024-01-15T22:14:05.197 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-15T22:14:05.209 INFO:teuthology.task.internal:Saving configuration 2024-01-15T22:14:05.222 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-15T22:14:05.229 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-15T22:14:05.255 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi099.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-15_22:02:29-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7517238', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-15 22:04:58.502999', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMot5Qaeq01nq2YXbQR2TDuQS4ijbfr9CnnRbMpJBClwb7TSaR9s+biN/qOmzGjqSKwc0rJptpRmHgw32mRGdUE='} 2024-01-15T22:14:05.277 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi162.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-15_22:02:29-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7517238', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-15 22:04:58.501950', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE9yp1RriOwRgcLJDD6X2eivuvf40mVddiFtyoo0tvH3VNXb4FgLO/0b5OsnIpZcsKUUvRfKEaq6ONa3eaRZJIc='} 2024-01-15T22:14:05.277 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-15T22:14:05.283 INFO:teuthology.task.internal:roles: ubuntu@smithi099.front.sepia.ceph.com - ['host.a', 'client.0'] 2024-01-15T22:14:05.284 INFO:teuthology.task.internal:roles: ubuntu@smithi162.front.sepia.ceph.com - ['host.b', 'client.1'] 2024-01-15T22:14:05.284 INFO:teuthology.run_tasks:Running task console_log... 2024-01-15T22:14:05.361 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fdd70d05ee0>, signals=[15]) 2024-01-15T22:14:05.361 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-15T22:14:05.367 INFO:teuthology.task.internal:Opening connections... 2024-01-15T22:14:05.368 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi099.front.sepia.ceph.com 2024-01-15T22:14:05.369 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T22:14:05.492 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi162.front.sepia.ceph.com 2024-01-15T22:14:05.493 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T22:14:05.575 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-15T22:14:05.581 DEBUG:teuthology.orchestra.run.smithi099:> uname -m 2024-01-15T22:14:05.602 INFO:teuthology.orchestra.run.smithi099.stdout:x86_64 2024-01-15T22:14:05.603 DEBUG:teuthology.orchestra.run.smithi099:> cat /etc/os-release 2024-01-15T22:14:05.663 INFO:teuthology.orchestra.run.smithi099.stdout:NAME="Red Hat Enterprise Linux" 2024-01-15T22:14:05.663 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION="8.6 (Ootpa)" 2024-01-15T22:14:05.663 INFO:teuthology.orchestra.run.smithi099.stdout:ID="rhel" 2024-01-15T22:14:05.663 INFO:teuthology.orchestra.run.smithi099.stdout:ID_LIKE="fedora" 2024-01-15T22:14:05.663 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION_ID="8.6" 2024-01-15T22:14:05.663 INFO:teuthology.orchestra.run.smithi099.stdout:PLATFORM_ID="platform:el8" 2024-01-15T22:14:05.663 INFO:teuthology.orchestra.run.smithi099.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-01-15T22:14:05.663 INFO:teuthology.orchestra.run.smithi099.stdout:ANSI_COLOR="0;31" 2024-01-15T22:14:05.663 INFO:teuthology.orchestra.run.smithi099.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-01-15T22:14:05.664 INFO:teuthology.orchestra.run.smithi099.stdout:HOME_URL="https://www.redhat.com/" 2024-01-15T22:14:05.664 INFO:teuthology.orchestra.run.smithi099.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-01-15T22:14:05.664 INFO:teuthology.orchestra.run.smithi099.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-15T22:14:05.664 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:14:05.664 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-15T22:14:05.664 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-01-15T22:14:05.664 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-01-15T22:14:05.664 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-01-15T22:14:05.665 INFO:teuthology.lock.ops:Updating smithi099.front.sepia.ceph.com on lock server 2024-01-15T22:14:05.693 DEBUG:teuthology.orchestra.run.smithi162:> uname -m 2024-01-15T22:14:05.717 INFO:teuthology.orchestra.run.smithi162.stdout:x86_64 2024-01-15T22:14:05.717 DEBUG:teuthology.orchestra.run.smithi162:> cat /etc/os-release 2024-01-15T22:14:05.778 INFO:teuthology.orchestra.run.smithi162.stdout:NAME="Red Hat Enterprise Linux" 2024-01-15T22:14:05.778 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION="8.6 (Ootpa)" 2024-01-15T22:14:05.778 INFO:teuthology.orchestra.run.smithi162.stdout:ID="rhel" 2024-01-15T22:14:05.778 INFO:teuthology.orchestra.run.smithi162.stdout:ID_LIKE="fedora" 2024-01-15T22:14:05.778 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION_ID="8.6" 2024-01-15T22:14:05.778 INFO:teuthology.orchestra.run.smithi162.stdout:PLATFORM_ID="platform:el8" 2024-01-15T22:14:05.778 INFO:teuthology.orchestra.run.smithi162.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-01-15T22:14:05.778 INFO:teuthology.orchestra.run.smithi162.stdout:ANSI_COLOR="0;31" 2024-01-15T22:14:05.778 INFO:teuthology.orchestra.run.smithi162.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-01-15T22:14:05.779 INFO:teuthology.orchestra.run.smithi162.stdout:HOME_URL="https://www.redhat.com/" 2024-01-15T22:14:05.779 INFO:teuthology.orchestra.run.smithi162.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-01-15T22:14:05.779 INFO:teuthology.orchestra.run.smithi162.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-15T22:14:05.779 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-15T22:14:05.779 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-15T22:14:05.779 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-01-15T22:14:05.779 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-01-15T22:14:05.779 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-01-15T22:14:05.779 INFO:teuthology.lock.ops:Updating smithi162.front.sepia.ceph.com on lock server 2024-01-15T22:14:05.801 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-15T22:14:05.809 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-15T22:14:05.894 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-15T22:14:05.894 DEBUG:teuthology.orchestra.run.smithi099:> test '!' -e /home/ubuntu/cephtest 2024-01-15T22:14:05.896 DEBUG:teuthology.orchestra.run.smithi162:> test '!' -e /home/ubuntu/cephtest 2024-01-15T22:14:05.916 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-15T22:14:05.922 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-15T22:14:05.922 DEBUG:teuthology.orchestra.run.smithi099:> test -z $(ls -A /var/lib/ceph) 2024-01-15T22:14:05.957 DEBUG:teuthology.orchestra.run.smithi162:> test -z $(ls -A /var/lib/ceph) 2024-01-15T22:14:05.984 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-15T22:14:06.038 INFO:teuthology.run_tasks:Running task kernel... 2024-01-15T22:14:06.053 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-01-15T22:14:06.053 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-01-15T22:14:06.054 DEBUG:teuthology.orchestra.run.smithi099:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-15T22:14:06.054 DEBUG:teuthology.orchestra.run.smithi162:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-15T22:14:06.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T22:14:06.073 DEBUG:teuthology.orchestra.run.smithi099:> uname -r 2024-01-15T22:14:06.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T22:14:06.075 DEBUG:teuthology.orchestra.run.smithi162:> uname -r 2024-01-15T22:14:06.134 INFO:teuthology.orchestra.run.smithi162.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-01-15T22:14:06.134 INFO:teuthology.task.kernel:Running kernel on smithi162: 4.18.0-513.9.1.el8_9.x86_64 2024-01-15T22:14:06.134 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum install -y kernel 2024-01-15T22:14:06.136 INFO:teuthology.orchestra.run.smithi099.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-01-15T22:14:06.136 INFO:teuthology.task.kernel:Running kernel on smithi099: 4.18.0-513.9.1.el8_9.x86_64 2024-01-15T22:14:06.136 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum install -y kernel 2024-01-15T22:14:07.638 INFO:teuthology.orchestra.run.smithi162.stdout:Updating Subscription Management repositories. 2024-01-15T22:14:07.638 INFO:teuthology.orchestra.run.smithi162.stdout:Unable to read consumer identity 2024-01-15T22:14:07.658 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-01-15T22:14:07.659 INFO:teuthology.orchestra.run.smithi099.stdout:Unable to read consumer identity 2024-01-15T22:14:07.758 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:14:07.758 INFO:teuthology.orchestra.run.smithi099.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-15T22:14:07.758 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:14:07.763 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-15T22:14:07.763 INFO:teuthology.orchestra.run.smithi162.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-15T22:14:07.763 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-15T22:14:09.461 INFO:teuthology.orchestra.run.smithi162.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2024-01-15T22:14:09.627 INFO:teuthology.orchestra.run.smithi099.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2024-01-15T22:14:10.455 INFO:teuthology.orchestra.run.smithi162.stdout:Extra Packages for Enterprise Linux 17 MB/s | 16 MB 00:00 2024-01-15T22:14:13.361 INFO:teuthology.orchestra.run.smithi099.stdout:Extra Packages for Enterprise Linux 4.4 MB/s | 16 MB 00:03 2024-01-15T22:14:14.003 INFO:teuthology.orchestra.run.smithi162.stdout:lab-extras 619 kB/s | 24 kB 00:00 2024-01-15T22:14:14.739 INFO:teuthology.orchestra.run.smithi162.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-01-15T22:14:14.739 INFO:teuthology.orchestra.run.smithi162.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-01-15T22:14:14.829 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-15T22:14:14.830 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-01-15T22:14:14.830 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-15T22:14:14.968 DEBUG:teuthology.orchestra.run.smithi162:> echo no | sudo yum reinstall kernel || true 2024-01-15T22:14:15.309 INFO:teuthology.orchestra.run.smithi162.stdout:Updating Subscription Management repositories. 2024-01-15T22:14:15.310 INFO:teuthology.orchestra.run.smithi162.stdout:Unable to read consumer identity 2024-01-15T22:14:15.409 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-15T22:14:15.409 INFO:teuthology.orchestra.run.smithi162.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-15T22:14:15.409 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-15T22:14:15.723 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:02 ago on Mon 15 Jan 2024 10:14:13 PM UTC. 2024-01-15T22:14:15.802 INFO:teuthology.orchestra.run.smithi162.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-01-15T22:14:15.802 INFO:teuthology.orchestra.run.smithi162.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-01-15T22:14:15.839 INFO:teuthology.orchestra.run.smithi162.stderr:Error: No packages marked for reinstall. 2024-01-15T22:14:15.888 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum reinstall -y kernel || true 2024-01-15T22:14:16.226 INFO:teuthology.orchestra.run.smithi162.stdout:Updating Subscription Management repositories. 2024-01-15T22:14:16.226 INFO:teuthology.orchestra.run.smithi162.stdout:Unable to read consumer identity 2024-01-15T22:14:16.309 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-15T22:14:16.309 INFO:teuthology.orchestra.run.smithi162.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-15T22:14:16.309 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-15T22:14:16.623 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:03 ago on Mon 15 Jan 2024 10:14:13 PM UTC. 2024-01-15T22:14:16.689 INFO:teuthology.orchestra.run.smithi162.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-01-15T22:14:16.689 INFO:teuthology.orchestra.run.smithi162.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-01-15T22:14:16.719 INFO:teuthology.orchestra.run.smithi162.stderr:Error: No packages marked for reinstall. 2024-01-15T22:14:16.772 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q kernel | sort -rV | head -n 1 2024-01-15T22:14:16.936 INFO:teuthology.orchestra.run.smithi162.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-01-15T22:14:16.936 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-01-15T22:14:16.936 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-01-15T22:14:16.936 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-01-15T22:14:16.936 DEBUG:teuthology.orchestra.run.smithi162:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-15T22:14:16.944 INFO:teuthology.orchestra.run.smithi099.stdout:lab-extras 431 kB/s | 24 kB 00:00 2024-01-15T22:14:16.973 INFO:teuthology.orchestra.run.smithi162.stdout:ttyS1 2024-01-15T22:14:16.986 DEBUG:teuthology.parallel:result is None 2024-01-15T22:14:17.704 INFO:teuthology.orchestra.run.smithi099.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-01-15T22:14:17.705 INFO:teuthology.orchestra.run.smithi099.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-01-15T22:14:17.787 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-01-15T22:14:17.788 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-01-15T22:14:17.788 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-01-15T22:14:17.906 DEBUG:teuthology.orchestra.run.smithi099:> echo no | sudo yum reinstall kernel || true 2024-01-15T22:14:18.242 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-01-15T22:14:18.242 INFO:teuthology.orchestra.run.smithi099.stdout:Unable to read consumer identity 2024-01-15T22:14:18.334 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:14:18.334 INFO:teuthology.orchestra.run.smithi099.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-15T22:14:18.334 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:14:18.649 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:00:02 ago on Mon 15 Jan 2024 10:14:16 PM UTC. 2024-01-15T22:14:18.722 INFO:teuthology.orchestra.run.smithi099.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-01-15T22:14:18.729 INFO:teuthology.orchestra.run.smithi099.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-01-15T22:14:18.768 INFO:teuthology.orchestra.run.smithi099.stderr:Error: No packages marked for reinstall. 2024-01-15T22:14:18.815 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum reinstall -y kernel || true 2024-01-15T22:14:19.149 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-01-15T22:14:19.149 INFO:teuthology.orchestra.run.smithi099.stdout:Unable to read consumer identity 2024-01-15T22:14:19.217 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:14:19.217 INFO:teuthology.orchestra.run.smithi099.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-15T22:14:19.217 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:14:19.524 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:00:03 ago on Mon 15 Jan 2024 10:14:16 PM UTC. 2024-01-15T22:14:19.590 INFO:teuthology.orchestra.run.smithi099.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-01-15T22:14:19.590 INFO:teuthology.orchestra.run.smithi099.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-01-15T22:14:19.623 INFO:teuthology.orchestra.run.smithi099.stderr:Error: No packages marked for reinstall. 2024-01-15T22:14:19.669 DEBUG:teuthology.orchestra.run.smithi099:> rpm -q kernel | sort -rV | head -n 1 2024-01-15T22:14:19.844 INFO:teuthology.orchestra.run.smithi099.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-01-15T22:14:19.844 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-01-15T22:14:19.844 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-01-15T22:14:19.844 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-01-15T22:14:19.845 DEBUG:teuthology.orchestra.run.smithi099:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-15T22:14:19.884 INFO:teuthology.orchestra.run.smithi099.stdout:ttyS1 2024-01-15T22:14:19.903 DEBUG:teuthology.parallel:result is None 2024-01-15T22:14:19.903 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-15T22:14:19.910 INFO:teuthology.task.internal:Creating test directory... 2024-01-15T22:14:19.911 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-15T22:14:19.945 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-15T22:14:19.966 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-15T22:14:19.972 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-15T22:14:19.979 INFO:teuthology.task.internal:Creating archive directory... 2024-01-15T22:14:19.980 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-15T22:14:20.002 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-15T22:14:20.037 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-15T22:14:20.044 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-15T22:14:20.044 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-15T22:14:20.067 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-15T22:14:20.106 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-15T22:14:20.109 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-15T22:14:20.118 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-15T22:14:20.119 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-15T22:14:20.121 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-15T22:14:20.128 INFO:teuthology.task.internal:Configuring sudo... 2024-01-15T22:14:20.128 DEBUG:teuthology.orchestra.run.smithi099:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-15T22:14:20.130 DEBUG:teuthology.orchestra.run.smithi162:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-15T22:14:20.190 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-15T22:14:20.201 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-15T22:14:20.201 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-15T22:14:20.203 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-15T22:14:20.249 DEBUG:teuthology.orchestra.run.smithi099:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-15T22:14:20.302 DEBUG:teuthology.orchestra.run.smithi099:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-15T22:14:20.349 DEBUG:teuthology.orchestra.run.smithi099:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-15T22:14:20.435 DEBUG:teuthology.orchestra.run.smithi099:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-15T22:14:20.475 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-01-15T22:14:20.475 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-15T22:14:20.542 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-15T22:14:20.588 DEBUG:teuthology.orchestra.run.smithi162:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-15T22:14:20.649 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-15T22:14:20.736 DEBUG:teuthology.orchestra.run.smithi162:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-15T22:14:20.779 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-01-15T22:14:20.779 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-15T22:14:20.848 DEBUG:teuthology.orchestra.run.smithi099:> sudo service rsyslog restart 2024-01-15T22:14:20.851 DEBUG:teuthology.orchestra.run.smithi162:> sudo service rsyslog restart 2024-01-15T22:14:20.897 INFO:teuthology.orchestra.run.smithi099.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-15T22:14:20.994 INFO:teuthology.orchestra.run.smithi162.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-15T22:14:21.201 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-15T22:14:21.211 INFO:teuthology.task.internal:Starting timer... 2024-01-15T22:14:21.211 INFO:teuthology.run_tasks:Running task pcp... 2024-01-15T22:14:21.222 INFO:teuthology.run_tasks:Running task selinux... 2024-01-15T22:14:21.231 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-01-15T22:14:21.231 DEBUG:teuthology.orchestra.run.smithi099:> sudo service auditd rotate 2024-01-15T22:14:21.292 INFO:teuthology.orchestra.run.smithi099.stdout:Rotating logs: 2024-01-15T22:14:21.294 DEBUG:teuthology.orchestra.run.smithi162:> sudo service auditd rotate 2024-01-15T22:14:21.355 INFO:teuthology.orchestra.run.smithi162.stdout:Rotating logs: 2024-01-15T22:14:21.356 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-15T22:14:21.357 DEBUG:teuthology.orchestra.run.smithi099:> /usr/sbin/getenforce 2024-01-15T22:14:21.381 INFO:teuthology.orchestra.run.smithi099.stdout:Permissive 2024-01-15T22:14:21.381 DEBUG:teuthology.orchestra.run.smithi162:> /usr/sbin/getenforce 2024-01-15T22:14:21.400 INFO:teuthology.orchestra.run.smithi162.stdout:Permissive 2024-01-15T22:14:21.402 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi099.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi162.front.sepia.ceph.com': 'permissive'} 2024-01-15T22:14:21.402 DEBUG:teuthology.orchestra.run.smithi099:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-15T22:14:21.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T22:14:21.450 DEBUG:teuthology.orchestra.run.smithi162:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-15T22:14:21.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T22:14:21.477 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-15T22:14:21.478 DEBUG:teuthology.orchestra.run.smithi099:> sudo /usr/sbin/setenforce permissive 2024-01-15T22:14:21.527 DEBUG:teuthology.orchestra.run.smithi162:> sudo /usr/sbin/setenforce permissive 2024-01-15T22:14:21.554 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-15T22:14:21.564 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-15T22:14:21.581 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-15T22:14:21.583 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi099.front.sepia.ceph.com,smithi162.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-15T22:23:27.241 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi099.front.sepia.ceph.com'), Remote(name='ubuntu@smithi162.front.sepia.ceph.com')] 2024-01-15T22:23:27.242 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-01-15T22:23:27.243 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T22:23:27.317 DEBUG:teuthology.orchestra.run.smithi099:> true 2024-01-15T22:23:27.389 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-01-15T22:23:27.389 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-01-15T22:23:27.390 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T22:23:27.465 DEBUG:teuthology.orchestra.run.smithi162:> true 2024-01-15T22:23:27.538 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-01-15T22:23:27.538 INFO:teuthology.run_tasks:Running task clock... 2024-01-15T22:23:27.548 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-15T22:23:27.548 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-15T22:23:27.549 DEBUG:teuthology.orchestra.run.smithi099:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-15T22:23:27.551 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-15T22:23:27.552 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-15T22:23:27.582 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-15T22:23:27.597 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-15T22:23:27.611 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-15T22:23:27.626 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-15T22:23:27.670 INFO:teuthology.orchestra.run.smithi099.stderr:sudo: ntpd: command not found 2024-01-15T22:23:27.682 INFO:teuthology.orchestra.run.smithi099.stdout:506 Cannot talk to daemon 2024-01-15T22:23:27.696 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-15T22:23:27.698 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: ntpd: command not found 2024-01-15T22:23:27.710 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-15T22:23:27.711 INFO:teuthology.orchestra.run.smithi162.stdout:506 Cannot talk to daemon 2024-01-15T22:23:27.725 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-15T22:23:27.739 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-15T22:23:27.788 INFO:teuthology.orchestra.run.smithi099.stderr:bash: ntpq: command not found 2024-01-15T22:23:27.791 INFO:teuthology.orchestra.run.smithi099.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-15T22:23:27.791 INFO:teuthology.orchestra.run.smithi099.stdout:=============================================================================== 2024-01-15T22:23:27.791 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-15T22:23:27.791 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-15T22:23:27.791 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-15T22:23:27.791 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-15T22:23:27.820 INFO:teuthology.orchestra.run.smithi162.stderr:bash: ntpq: command not found 2024-01-15T22:23:27.824 INFO:teuthology.orchestra.run.smithi162.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-15T22:23:27.824 INFO:teuthology.orchestra.run.smithi162.stdout:=============================================================================== 2024-01-15T22:23:27.824 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-15T22:23:27.824 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-15T22:23:27.824 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-15T22:23:27.824 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-15T22:23:27.825 INFO:teuthology.run_tasks:Running task pexec... 2024-01-15T22:23:27.834 INFO:teuthology.task.pexec:Executing custom commands... 2024-01-15T22:23:27.834 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi099.front.sepia.ceph.com 2024-01-15T22:23:27.835 DEBUG:teuthology.orchestra.run.smithi099:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-15T22:23:27.835 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi162.front.sepia.ceph.com 2024-01-15T22:23:27.835 DEBUG:teuthology.orchestra.run.smithi162:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-15T22:23:28.210 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-01-15T22:23:28.242 INFO:teuthology.orchestra.run.smithi162.stdout:Updating Subscription Management repositories. 2024-01-15T22:23:30.107 INFO:teuthology.orchestra.run.smithi162.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 34 kB/s | 4.5 kB 00:00 2024-01-15T22:23:30.295 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 18 kB/s | 4.1 kB 00:00 2024-01-15T22:23:30.662 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 42 kB/s | 4.5 kB 00:00 2024-01-15T22:23:30.662 INFO:teuthology.orchestra.run.smithi162.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 42 kB/s | 4.1 kB 00:00 2024-01-15T22:23:33.469 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-15T22:23:33.470 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-15T22:23:33.470 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2024-01-15T22:23:33.470 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-15T22:23:33.470 INFO:teuthology.orchestra.run.smithi162.stdout:Resetting modules: 2024-01-15T22:23:33.470 INFO:teuthology.orchestra.run.smithi162.stdout: container-tools 2024-01-15T22:23:33.470 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-15T22:23:33.470 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-15T22:23:33.470 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-01-15T22:23:33.471 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-15T22:23:33.696 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-01-15T22:23:33.697 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-01-15T22:23:33.697 INFO:teuthology.orchestra.run.smithi099.stdout: Package Architecture Version Repository Size 2024-01-15T22:23:33.697 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-01-15T22:23:33.697 INFO:teuthology.orchestra.run.smithi099.stdout:Resetting modules: 2024-01-15T22:23:33.697 INFO:teuthology.orchestra.run.smithi099.stdout: container-tools 2024-01-15T22:23:33.697 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:23:33.697 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-01-15T22:23:33.697 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-01-15T22:23:33.697 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:23:47.367 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-15T22:23:47.672 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-01-15T22:23:47.831 INFO:teuthology.orchestra.run.smithi162.stdout:Updating Subscription Management repositories. 2024-01-15T22:23:48.127 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-01-15T22:23:49.742 INFO:teuthology.orchestra.run.smithi162.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 27 kB/s | 4.5 kB 00:00 2024-01-15T22:23:49.910 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 41 kB/s | 4.1 kB 00:00 2024-01-15T22:23:50.338 INFO:teuthology.orchestra.run.smithi162.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 27 kB/s | 4.1 kB 00:00 2024-01-15T22:23:50.350 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 24 kB/s | 4.5 kB 00:00 2024-01-15T22:23:53.116 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-01-15T22:23:53.254 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-01-15T22:23:53.271 INFO:teuthology.orchestra.run.smithi162.stdout:========================================================================================================================== 2024-01-15T22:23:53.271 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-01-15T22:23:53.271 INFO:teuthology.orchestra.run.smithi162.stdout:========================================================================================================================== 2024-01-15T22:23:53.271 INFO:teuthology.orchestra.run.smithi162.stdout:Installing group/module packages: 2024-01-15T22:23:53.271 INFO:teuthology.orchestra.run.smithi162.stdout: buildah x86_64 1.19.9-6.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 6.6 M 2024-01-15T22:23:53.272 INFO:teuthology.orchestra.run.smithi162.stdout: cockpit-podman noarch 29-2.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 1.1 M 2024-01-15T22:23:53.272 INFO:teuthology.orchestra.run.smithi162.stdout: crun x86_64 0.18-3.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 185 k 2024-01-15T22:23:53.272 INFO:teuthology.orchestra.run.smithi162.stdout: skopeo x86_64 1:1.2.4-2.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 6.2 M 2024-01-15T22:23:53.272 INFO:teuthology.orchestra.run.smithi162.stdout: toolbox x86_64 0.0.99.3-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 2.2 M 2024-01-15T22:23:53.272 INFO:teuthology.orchestra.run.smithi162.stdout: udica noarch 0.2.4-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 51 k 2024-01-15T22:23:53.272 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-01-15T22:23:53.272 INFO:teuthology.orchestra.run.smithi162.stdout: yajl x86_64 2.1.0-12.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2024-01-15T22:23:53.272 INFO:teuthology.orchestra.run.smithi162.stdout:Downgrading: 2024-01-15T22:23:53.272 INFO:teuthology.orchestra.run.smithi162.stdout: conmon x86_64 2:2.0.26-3.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 51 k 2024-01-15T22:23:53.272 INFO:teuthology.orchestra.run.smithi162.stdout: container-selinux noarch 2:2.189.0-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 55 k 2024-01-15T22:23:53.272 INFO:teuthology.orchestra.run.smithi162.stdout: containernetworking-plugins x86_64 0.9.1-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 19 M 2024-01-15T22:23:53.272 INFO:teuthology.orchestra.run.smithi162.stdout: containers-common x86_64 1:1.2.4-2.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 91 k 2024-01-15T22:23:53.272 INFO:teuthology.orchestra.run.smithi162.stdout: criu x86_64 3.15-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 511 k 2024-01-15T22:23:53.273 INFO:teuthology.orchestra.run.smithi162.stdout: fuse-overlayfs x86_64 1.4.0-2.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 72 k 2024-01-15T22:23:53.273 INFO:teuthology.orchestra.run.smithi162.stdout: libslirp x86_64 4.3.1-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 69 k 2024-01-15T22:23:53.273 INFO:teuthology.orchestra.run.smithi162.stdout: podman x86_64 3.0.1-15.module+el8.7.0+18021+1e286cc8 rhel-8-for-x86_64-appstream-rpms 11 M 2024-01-15T22:23:53.273 INFO:teuthology.orchestra.run.smithi162.stdout: podman-catatonit x86_64 3.0.1-15.module+el8.7.0+18021+1e286cc8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-01-15T22:23:53.273 INFO:teuthology.orchestra.run.smithi162.stdout: podman-docker noarch 3.0.1-15.module+el8.7.0+18021+1e286cc8 rhel-8-for-x86_64-appstream-rpms 57 k 2024-01-15T22:23:53.273 INFO:teuthology.orchestra.run.smithi162.stdout: runc x86_64 1.0.0-73.rc95.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 2.9 M 2024-01-15T22:23:53.273 INFO:teuthology.orchestra.run.smithi162.stdout: slirp4netns x86_64 1.1.8-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 51 k 2024-01-15T22:23:53.273 INFO:teuthology.orchestra.run.smithi162.stdout:Installing module profiles: 2024-01-15T22:23:53.273 INFO:teuthology.orchestra.run.smithi162.stdout: container-tools/common 2024-01-15T22:23:53.273 INFO:teuthology.orchestra.run.smithi162.stdout:Enabling module streams: 2024-01-15T22:23:53.273 INFO:teuthology.orchestra.run.smithi162.stdout: container-tools 3.0 2024-01-15T22:23:53.273 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-15T22:23:53.273 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-01-15T22:23:53.273 INFO:teuthology.orchestra.run.smithi162.stdout:========================================================================================================================== 2024-01-15T22:23:53.274 INFO:teuthology.orchestra.run.smithi162.stdout:Install 7 Packages 2024-01-15T22:23:53.274 INFO:teuthology.orchestra.run.smithi162.stdout:Downgrade 12 Packages 2024-01-15T22:23:53.274 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-15T22:23:53.274 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 50 M 2024-01-15T22:23:53.274 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-01-15T22:23:53.405 INFO:teuthology.orchestra.run.smithi099.stdout:========================================================================================================================== 2024-01-15T22:23:53.405 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-01-15T22:23:53.405 INFO:teuthology.orchestra.run.smithi099.stdout:========================================================================================================================== 2024-01-15T22:23:53.405 INFO:teuthology.orchestra.run.smithi099.stdout:Installing group/module packages: 2024-01-15T22:23:53.405 INFO:teuthology.orchestra.run.smithi099.stdout: buildah x86_64 1.19.9-6.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 6.6 M 2024-01-15T22:23:53.406 INFO:teuthology.orchestra.run.smithi099.stdout: cockpit-podman noarch 29-2.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 1.1 M 2024-01-15T22:23:53.406 INFO:teuthology.orchestra.run.smithi099.stdout: crun x86_64 0.18-3.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 185 k 2024-01-15T22:23:53.406 INFO:teuthology.orchestra.run.smithi099.stdout: skopeo x86_64 1:1.2.4-2.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 6.2 M 2024-01-15T22:23:53.406 INFO:teuthology.orchestra.run.smithi099.stdout: toolbox x86_64 0.0.99.3-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 2.2 M 2024-01-15T22:23:53.406 INFO:teuthology.orchestra.run.smithi099.stdout: udica noarch 0.2.4-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 51 k 2024-01-15T22:23:53.406 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-01-15T22:23:53.406 INFO:teuthology.orchestra.run.smithi099.stdout: yajl x86_64 2.1.0-12.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2024-01-15T22:23:53.406 INFO:teuthology.orchestra.run.smithi099.stdout:Downgrading: 2024-01-15T22:23:53.406 INFO:teuthology.orchestra.run.smithi099.stdout: conmon x86_64 2:2.0.26-3.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 51 k 2024-01-15T22:23:53.406 INFO:teuthology.orchestra.run.smithi099.stdout: container-selinux noarch 2:2.189.0-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 55 k 2024-01-15T22:23:53.406 INFO:teuthology.orchestra.run.smithi099.stdout: containernetworking-plugins x86_64 0.9.1-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 19 M 2024-01-15T22:23:53.406 INFO:teuthology.orchestra.run.smithi099.stdout: containers-common x86_64 1:1.2.4-2.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 91 k 2024-01-15T22:23:53.406 INFO:teuthology.orchestra.run.smithi099.stdout: criu x86_64 3.15-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 511 k 2024-01-15T22:23:53.407 INFO:teuthology.orchestra.run.smithi099.stdout: fuse-overlayfs x86_64 1.4.0-2.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 72 k 2024-01-15T22:23:53.407 INFO:teuthology.orchestra.run.smithi099.stdout: libslirp x86_64 4.3.1-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 69 k 2024-01-15T22:23:53.407 INFO:teuthology.orchestra.run.smithi099.stdout: podman x86_64 3.0.1-15.module+el8.7.0+18021+1e286cc8 rhel-8-for-x86_64-appstream-rpms 11 M 2024-01-15T22:23:53.407 INFO:teuthology.orchestra.run.smithi099.stdout: podman-catatonit x86_64 3.0.1-15.module+el8.7.0+18021+1e286cc8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-01-15T22:23:53.407 INFO:teuthology.orchestra.run.smithi099.stdout: podman-docker noarch 3.0.1-15.module+el8.7.0+18021+1e286cc8 rhel-8-for-x86_64-appstream-rpms 57 k 2024-01-15T22:23:53.407 INFO:teuthology.orchestra.run.smithi099.stdout: runc x86_64 1.0.0-73.rc95.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 2.9 M 2024-01-15T22:23:53.407 INFO:teuthology.orchestra.run.smithi099.stdout: slirp4netns x86_64 1.1.8-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 51 k 2024-01-15T22:23:53.407 INFO:teuthology.orchestra.run.smithi099.stdout:Installing module profiles: 2024-01-15T22:23:53.407 INFO:teuthology.orchestra.run.smithi099.stdout: container-tools/common 2024-01-15T22:23:53.407 INFO:teuthology.orchestra.run.smithi099.stdout:Enabling module streams: 2024-01-15T22:23:53.407 INFO:teuthology.orchestra.run.smithi099.stdout: container-tools 3.0 2024-01-15T22:23:53.407 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:23:53.407 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-01-15T22:23:53.407 INFO:teuthology.orchestra.run.smithi099.stdout:========================================================================================================================== 2024-01-15T22:23:53.407 INFO:teuthology.orchestra.run.smithi099.stdout:Install 7 Packages 2024-01-15T22:23:53.407 INFO:teuthology.orchestra.run.smithi099.stdout:Downgrade 12 Packages 2024-01-15T22:23:53.408 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:23:53.408 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 50 M 2024-01-15T22:23:53.408 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-01-15T22:23:53.439 INFO:teuthology.orchestra.run.smithi162.stdout:(1/19): conmon-2.0.26-3.module+el8.7.0+16533+44 312 kB/s | 51 kB 00:00 2024-01-15T22:23:53.497 INFO:teuthology.orchestra.run.smithi162.stdout:(2/19): criu-3.15-1.module+el8.7.0+16533+44634e 2.2 MB/s | 511 kB 00:00 2024-01-15T22:23:53.531 INFO:teuthology.orchestra.run.smithi162.stdout:(3/19): fuse-overlayfs-1.4.0-2.module+el8.7.0+1 782 kB/s | 72 kB 00:00 2024-01-15T22:23:53.544 INFO:teuthology.orchestra.run.smithi099.stdout:(1/19): conmon-2.0.26-3.module+el8.7.0+16533+44 378 kB/s | 51 kB 00:00 2024-01-15T22:23:53.586 INFO:teuthology.orchestra.run.smithi099.stdout:(2/19): criu-3.15-1.module+el8.7.0+16533+44634e 2.8 MB/s | 511 kB 00:00 2024-01-15T22:23:53.686 INFO:teuthology.orchestra.run.smithi099.stdout:(3/19): fuse-overlayfs-1.4.0-2.module+el8.7.0+1 507 kB/s | 72 kB 00:00 2024-01-15T22:23:53.706 INFO:teuthology.orchestra.run.smithi162.stdout:(4/19): libslirp-4.3.1-1.module+el8.7.0+16533+4 331 kB/s | 69 kB 00:00 2024-01-15T22:23:53.711 INFO:teuthology.orchestra.run.smithi099.stdout:(4/19): libslirp-4.3.1-1.module+el8.7.0+16533+4 551 kB/s | 69 kB 00:00 2024-01-15T22:23:53.940 INFO:teuthology.orchestra.run.smithi162.stdout:(5/19): containernetworking-plugins-0.9.1-1.mod 28 MB/s | 19 MB 00:00 2024-01-15T22:23:53.957 INFO:teuthology.orchestra.run.smithi162.stdout:(6/19): podman-docker-3.0.1-15.module+el8.7.0+1 228 kB/s | 57 kB 00:00 2024-01-15T22:23:54.012 INFO:teuthology.orchestra.run.smithi099.stdout:(5/19): containernetworking-plugins-0.9.1-1.mod 31 MB/s | 19 MB 00:00 2024-01-15T22:23:54.015 INFO:teuthology.orchestra.run.smithi162.stdout:(7/19): container-selinux-2.189.0-1.module+el8. 942 kB/s | 55 kB 00:00 2024-01-15T22:23:54.029 INFO:teuthology.orchestra.run.smithi099.stdout:(6/19): podman-docker-3.0.1-15.module+el8.7.0+1 180 kB/s | 57 kB 00:00 2024-01-15T22:23:54.150 INFO:teuthology.orchestra.run.smithi162.stdout:(8/19): podman-3.0.1-15.module+el8.7.0+18021+1e 17 MB/s | 11 MB 00:00 2024-01-15T22:23:54.154 INFO:teuthology.orchestra.run.smithi099.stdout:(7/19): runc-1.0.0-73.rc95.module+el8.7.0+16533 20 MB/s | 2.9 MB 00:00 2024-01-15T22:23:54.175 INFO:teuthology.orchestra.run.smithi162.stdout:(9/19): containers-common-1.2.4-2.module+el8.7. 575 kB/s | 91 kB 00:00 2024-01-15T22:23:54.196 INFO:teuthology.orchestra.run.smithi099.stdout:(8/19): container-selinux-2.189.0-1.module+el8. 329 kB/s | 55 kB 00:00 2024-01-15T22:23:54.233 INFO:teuthology.orchestra.run.smithi162.stdout:(10/19): runc-1.0.0-73.rc95.module+el8.7.0+1653 9.8 MB/s | 2.9 MB 00:00 2024-01-15T22:23:54.317 INFO:teuthology.orchestra.run.smithi162.stdout:(11/19): podman-catatonit-3.0.1-15.module+el8.7 1.9 MB/s | 323 kB 00:00 2024-01-15T22:23:54.338 INFO:teuthology.orchestra.run.smithi099.stdout:(9/19): podman-3.0.1-15.module+el8.7.0+18021+1e 16 MB/s | 11 MB 00:00 2024-01-15T22:23:54.343 INFO:teuthology.orchestra.run.smithi162.stdout:(12/19): slirp4netns-1.1.8-1.module+el8.7.0+165 306 kB/s | 51 kB 00:00 2024-01-15T22:23:54.363 INFO:teuthology.orchestra.run.smithi099.stdout:(10/19): containers-common-1.2.4-2.module+el8.7 436 kB/s | 91 kB 00:00 2024-01-15T22:23:54.388 INFO:teuthology.orchestra.run.smithi099.stdout:(11/19): podman-catatonit-3.0.1-15.module+el8.7 1.6 MB/s | 323 kB 00:00 2024-01-15T22:23:54.417 INFO:teuthology.orchestra.run.smithi162.stdout:(13/19): cockpit-podman-29-2.module+el8.7.0+165 5.8 MB/s | 1.1 MB 00:00 2024-01-15T22:23:54.472 INFO:teuthology.orchestra.run.smithi099.stdout:(12/19): cockpit-podman-29-2.module+el8.7.0+165 9.8 MB/s | 1.1 MB 00:00 2024-01-15T22:23:54.476 INFO:teuthology.orchestra.run.smithi162.stdout:(14/19): crun-0.18-3.module+el8.7.0+16533+44634 1.1 MB/s | 185 kB 00:00 2024-01-15T22:23:54.489 INFO:teuthology.orchestra.run.smithi099.stdout:(13/19): slirp4netns-1.1.8-1.module+el8.7.0+165 340 kB/s | 51 kB 00:00 2024-01-15T22:23:54.514 INFO:teuthology.orchestra.run.smithi099.stdout:(14/19): crun-0.18-3.module+el8.7.0+16533+44634 1.4 MB/s | 185 kB 00:00 2024-01-15T22:23:54.585 INFO:teuthology.orchestra.run.smithi162.stdout:(15/19): toolbox-0.0.99.3-1.module+el8.7.0+1653 13 MB/s | 2.2 MB 00:00 2024-01-15T22:23:54.614 INFO:teuthology.orchestra.run.smithi099.stdout:(15/19): toolbox-0.0.99.3-1.module+el8.7.0+1653 18 MB/s | 2.2 MB 00:00 2024-01-15T22:23:54.618 INFO:teuthology.orchestra.run.smithi162.stdout:(16/19): udica-0.2.4-1.module+el8.7.0+16533+446 357 kB/s | 51 kB 00:00 2024-01-15T22:23:54.702 INFO:teuthology.orchestra.run.smithi162.stdout:(17/19): skopeo-1.2.4-2.module+el8.7.0+16533+44 17 MB/s | 6.2 MB 00:00 2024-01-15T22:23:54.740 INFO:teuthology.orchestra.run.smithi099.stdout:(16/19): skopeo-1.2.4-2.module+el8.7.0+16533+44 23 MB/s | 6.2 MB 00:00 2024-01-15T22:23:54.764 INFO:teuthology.orchestra.run.smithi099.stdout:(17/19): udica-0.2.4-1.module+el8.7.0+16533+446 202 kB/s | 51 kB 00:00 2024-01-15T22:23:54.802 INFO:teuthology.orchestra.run.smithi162.stdout:(18/19): yajl-2.1.0-12.el8.x86_64.rpm 223 kB/s | 41 kB 00:00 2024-01-15T22:23:54.831 INFO:teuthology.orchestra.run.smithi099.stdout:(18/19): yajl-2.1.0-12.el8.x86_64.rpm 448 kB/s | 41 kB 00:00 2024-01-15T22:23:54.944 INFO:teuthology.orchestra.run.smithi162.stdout:(19/19): buildah-1.19.9-6.module+el8.7.0+16533+ 18 MB/s | 6.6 MB 00:00 2024-01-15T22:23:54.944 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-01-15T22:23:54.944 INFO:teuthology.orchestra.run.smithi162.stdout:Total 30 MB/s | 50 MB 00:01 2024-01-15T22:23:54.957 INFO:teuthology.orchestra.run.smithi099.stdout:(19/19): buildah-1.19.9-6.module+el8.7.0+16533+ 19 MB/s | 6.6 MB 00:00 2024-01-15T22:23:54.957 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-01-15T22:23:54.957 INFO:teuthology.orchestra.run.smithi099.stdout:Total 32 MB/s | 50 MB 00:01 2024-01-15T22:23:55.427 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-01-15T22:23:55.439 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-01-15T22:23:55.471 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-01-15T22:23:55.471 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-01-15T22:23:55.482 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-01-15T22:23:55.482 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-01-15T22:23:55.905 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-01-15T22:23:55.907 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-01-15T22:23:55.913 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-01-15T22:23:55.915 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-01-15T22:23:57.102 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-01-15T22:23:57.104 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-01-15T22:23:57.117 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: container-selinux-2:2.189.0-1.module+el8.7.0+16533+4 1/1 2024-01-15T22:23:57.120 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: container-selinux-2:2.189.0-1.module+el8.7.0+16533+4 1/1 2024-01-15T22:23:57.134 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: container-selinux-2:2.189.0-1.module+el8.7.0+16533 1/31 2024-01-15T22:23:57.137 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: container-selinux-2:2.189.0-1.module+el8.7.0+16533 1/31 2024-01-15T22:23:57.303 INFO:teuthology.orchestra.run.smithi162.stdout: Downgrading : container-selinux-2:2.189.0-1.module+el8.7.0+16533 1/31 2024-01-15T22:23:57.319 INFO:teuthology.orchestra.run.smithi099.stdout: Downgrading : container-selinux-2:2.189.0-1.module+el8.7.0+16533 1/31 2024-01-15T22:24:14.064 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: container-selinux-2:2.189.0-1.module+el8.7.0+16533 1/31 2024-01-15T22:24:14.273 INFO:teuthology.orchestra.run.smithi099.stdout: Downgrading : fuse-overlayfs-1.4.0-2.module+el8.7.0+16533+44634e 2/31 2024-01-15T22:24:14.284 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: container-selinux-2:2.189.0-1.module+el8.7.0+16533 1/31 2024-01-15T22:24:14.291 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: fuse-overlayfs-1.4.0-2.module+el8.7.0+16533+44634e 2/31 2024-01-15T22:24:14.432 INFO:teuthology.orchestra.run.smithi162.stdout: Downgrading : fuse-overlayfs-1.4.0-2.module+el8.7.0+16533+44634e 2/31 2024-01-15T22:24:14.450 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: fuse-overlayfs-1.4.0-2.module+el8.7.0+16533+44634e 2/31 2024-01-15T22:24:14.463 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : yajl-2.1.0-12.el8.x86_64 3/31 2024-01-15T22:24:14.622 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : yajl-2.1.0-12.el8.x86_64 3/31 2024-01-15T22:24:14.645 INFO:teuthology.orchestra.run.smithi099.stdout: Downgrading : libslirp-4.3.1-1.module+el8.7.0+16533+44634e24.x86 4/31 2024-01-15T22:24:14.796 INFO:teuthology.orchestra.run.smithi162.stdout: Downgrading : libslirp-4.3.1-1.module+el8.7.0+16533+44634e24.x86 4/31 2024-01-15T22:24:14.865 INFO:teuthology.orchestra.run.smithi099.stdout: Downgrading : slirp4netns-1.1.8-1.module+el8.7.0+16533+44634e24. 5/31 2024-01-15T22:24:14.865 INFO:teuthology.orchestra.run.smithi099.stdout: Downgrading : containers-common-1:1.2.4-2.module+el8.7.0+16533+4 6/31 2024-01-15T22:24:14.865 INFO:teuthology.orchestra.run.smithi099.stdout:warning: /etc/containers/registries.conf created as /etc/containers/registries.conf.rpmnew 2024-01-15T22:24:14.865 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:24:15.007 INFO:teuthology.orchestra.run.smithi162.stdout: Downgrading : slirp4netns-1.1.8-1.module+el8.7.0+16533+44634e24. 5/31 2024-01-15T22:24:15.007 INFO:teuthology.orchestra.run.smithi162.stdout: Downgrading : containers-common-1:1.2.4-2.module+el8.7.0+16533+4 6/31 2024-01-15T22:24:15.007 INFO:teuthology.orchestra.run.smithi162.stdout:warning: /etc/containers/registries.conf created as /etc/containers/registries.conf.rpmnew 2024-01-15T22:24:15.007 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-15T22:24:15.685 INFO:teuthology.orchestra.run.smithi099.stdout: Downgrading : criu-3.15-1.module+el8.7.0+16533+44634e24.x86_64 7/31 2024-01-15T22:24:15.739 INFO:teuthology.orchestra.run.smithi162.stdout: Downgrading : criu-3.15-1.module+el8.7.0+16533+44634e24.x86_64 7/31 2024-01-15T22:24:17.405 INFO:teuthology.orchestra.run.smithi099.stdout: Downgrading : runc-1.0.0-73.rc95.module+el8.7.0+16533+44634e24.x 8/31 2024-01-15T22:24:17.433 INFO:teuthology.orchestra.run.smithi162.stdout: Downgrading : runc-1.0.0-73.rc95.module+el8.7.0+16533+44634e24.x 8/31 2024-01-15T22:24:17.646 INFO:teuthology.orchestra.run.smithi162.stdout: Downgrading : containernetworking-plugins-0.9.1-1.module+el8.7.0 9/31 2024-01-15T22:24:17.687 INFO:teuthology.orchestra.run.smithi099.stdout: Downgrading : containernetworking-plugins-0.9.1-1.module+el8.7.0 9/31 2024-01-15T22:24:17.882 INFO:teuthology.orchestra.run.smithi162.stdout: Downgrading : conmon-2:2.0.26-3.module+el8.7.0+16533+44634e24.x8 10/31 2024-01-15T22:24:17.906 INFO:teuthology.orchestra.run.smithi099.stdout: Downgrading : conmon-2:2.0.26-3.module+el8.7.0+16533+44634e24.x8 10/31 2024-01-15T22:24:19.183 INFO:teuthology.orchestra.run.smithi162.stdout: Downgrading : podman-catatonit-3.0.1-15.module+el8.7.0+18021+1e2 11/31 2024-01-15T22:24:19.249 INFO:teuthology.orchestra.run.smithi099.stdout: Downgrading : podman-catatonit-3.0.1-15.module+el8.7.0+18021+1e2 11/31 2024-01-15T22:24:19.481 INFO:teuthology.orchestra.run.smithi162.stdout: Downgrading : podman-3.0.1-15.module+el8.7.0+18021+1e286cc8.x86_ 12/31 2024-01-15T22:24:19.565 INFO:teuthology.orchestra.run.smithi099.stdout: Downgrading : podman-3.0.1-15.module+el8.7.0+18021+1e286cc8.x86_ 12/31 2024-01-15T22:24:19.722 INFO:teuthology.orchestra.run.smithi162.stdout: Downgrading : podman-docker-3.0.1-15.module+el8.7.0+18021+1e286c 13/31 2024-01-15T22:24:19.805 INFO:teuthology.orchestra.run.smithi099.stdout: Downgrading : podman-docker-3.0.1-15.module+el8.7.0+18021+1e286c 13/31 2024-01-15T22:24:20.087 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : cockpit-podman-29-2.module+el8.7.0+16533+44634e24. 14/31 2024-01-15T22:24:20.179 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : cockpit-podman-29-2.module+el8.7.0+16533+44634e24. 14/31 2024-01-15T22:24:20.805 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : toolbox-0.0.99.3-1.module+el8.7.0+16533+44634e24.x 15/31 2024-01-15T22:24:20.894 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : toolbox-0.0.99.3-1.module+el8.7.0+16533+44634e24.x 15/31 2024-01-15T22:24:21.508 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : buildah-1.19.9-6.module+el8.7.0+16533+44634e24.x86 16/31 2024-01-15T22:24:21.609 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : buildah-1.19.9-6.module+el8.7.0+16533+44634e24.x86 16/31 2024-01-15T22:24:21.721 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : skopeo-1:1.2.4-2.module+el8.7.0+16533+44634e24.x86 17/31 2024-01-15T22:24:21.846 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : skopeo-1:1.2.4-2.module+el8.7.0+16533+44634e24.x86 17/31 2024-01-15T22:24:21.930 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : crun-0.18-3.module+el8.7.0+16533+44634e24.x86_64 18/31 2024-01-15T22:24:22.046 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : crun-0.18-3.module+el8.7.0+16533+44634e24.x86_64 18/31 2024-01-15T22:24:22.104 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : udica-0.2.4-1.module+el8.7.0+16533+44634e24.noarch 19/31 2024-01-15T22:24:22.232 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : podman-docker-3:4.6.1-4.module+el8.9.0+20326+38708 20/31 2024-01-15T22:24:22.245 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: podman-3:4.6.1-4.module+el8.9.0+20326+387084d0.x86 21/31 2024-01-15T22:24:22.245 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : podman-3:4.6.1-4.module+el8.9.0+20326+387084d0.x86 21/31 2024-01-15T22:24:22.261 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : udica-0.2.4-1.module+el8.7.0+16533+44634e24.noarch 19/31 2024-01-15T22:24:22.414 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : podman-docker-3:4.6.1-4.module+el8.9.0+20326+38708 20/31 2024-01-15T22:24:22.427 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: podman-3:4.6.1-4.module+el8.9.0+20326+387084d0.x86 21/31 2024-01-15T22:24:22.428 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : podman-3:4.6.1-4.module+el8.9.0+20326+387084d0.x86 21/31 2024-01-15T22:24:22.563 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: podman-3:4.6.1-4.module+el8.9.0+20326+387084d0.x86 21/31 2024-01-15T22:24:22.720 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : containers-common-2:1-70.module+el8.9.0+20326+3870 22/31 2024-01-15T22:24:22.720 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : container-selinux-2:2.221.0-1.module+el8.9.0+20326 23/31 2024-01-15T22:24:22.731 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: podman-3:4.6.1-4.module+el8.9.0+20326+387084d0.x86 21/31 2024-01-15T22:24:22.947 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : containers-common-2:1-70.module+el8.9.0+20326+3870 22/31 2024-01-15T22:24:22.947 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : container-selinux-2:2.221.0-1.module+el8.9.0+20326 23/31 2024-01-15T22:24:22.949 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: container-selinux-2:2.221.0-1.module+el8.9.0+20326 23/31 2024-01-15T22:24:22.949 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : podman-catatonit-3:4.6.1-4.module+el8.9.0+20326+38 24/31 2024-01-15T22:24:23.090 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: podman-catatonit-3:4.6.1-4.module+el8.9.0+20326+38 24/31 2024-01-15T22:24:23.132 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: container-selinux-2:2.221.0-1.module+el8.9.0+20326 23/31 2024-01-15T22:24:23.132 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : podman-catatonit-3:4.6.1-4.module+el8.9.0+20326+38 24/31 2024-01-15T22:24:23.282 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : slirp4netns-1.2.1-1.module+el8.9.0+20326+387084d0. 25/31 2024-01-15T22:24:23.289 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: podman-catatonit-3:4.6.1-4.module+el8.9.0+20326+38 24/31 2024-01-15T22:24:23.481 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : slirp4netns-1.2.1-1.module+el8.9.0+20326+387084d0. 25/31 2024-01-15T22:24:23.481 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : runc-1:1.1.9-1.module+el8.9.0+20326+387084d0.x86_6 26/31 2024-01-15T22:24:23.647 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : runc-1:1.1.9-1.module+el8.9.0+20326+387084d0.x86_6 26/31 2024-01-15T22:24:23.670 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : criu-3.18-4.module+el8.9.0+20326+387084d0.x86_64 27/31 2024-01-15T22:24:23.830 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : criu-3.18-4.module+el8.9.0+20326+387084d0.x86_64 27/31 2024-01-15T22:24:23.848 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : libslirp-4.4.0-1.module+el8.9.0+20326+387084d0.x86 28/31 2024-01-15T22:24:24.022 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : libslirp-4.4.0-1.module+el8.9.0+20326+387084d0.x86 28/31 2024-01-15T22:24:24.047 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : fuse-overlayfs-1.12-1.module+el8.9.0+20326+387084d 29/31 2024-01-15T22:24:24.206 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : fuse-overlayfs-1.12-1.module+el8.9.0+20326+387084d 29/31 2024-01-15T22:24:24.226 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : conmon-3:2.1.8-1.module+el8.9.0+20326+387084d0.x86 30/31 2024-01-15T22:24:24.226 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : containernetworking-plugins-1:1.3.0-4.module+el8.9 31/31 2024-01-15T22:24:24.375 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : conmon-3:2.1.8-1.module+el8.9.0+20326+387084d0.x86 30/31 2024-01-15T22:24:24.375 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : containernetworking-plugins-1:1.3.0-4.module+el8.9 31/31 2024-01-15T22:24:24.456 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: containernetworking-plugins-1:1.3.0-4.module+el8.9 31/31 2024-01-15T22:24:24.613 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: containernetworking-plugins-1:1.3.0-4.module+el8.9 31/31 2024-01-15T22:24:41.885 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: container-selinux-2:2.189.0-1.module+el8.7.0+16533 31/31 2024-01-15T22:24:42.126 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: container-selinux-2:2.189.0-1.module+el8.7.0+16533 31/31 2024-01-15T22:24:42.335 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: containernetworking-plugins-1:1.3.0-4.module+el8.9 31/31 2024-01-15T22:24:42.335 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : conmon-2:2.0.26-3.module+el8.7.0+16533+44634e24.x8 1/31 2024-01-15T22:24:42.335 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : conmon-3:2.1.8-1.module+el8.9.0+20326+387084d0.x86 2/31 2024-01-15T22:24:42.335 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : containernetworking-plugins-0.9.1-1.module+el8.7.0 3/31 2024-01-15T22:24:42.336 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : containernetworking-plugins-1:1.3.0-4.module+el8.9 4/31 2024-01-15T22:24:42.336 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : criu-3.15-1.module+el8.7.0+16533+44634e24.x86_64 5/31 2024-01-15T22:24:42.336 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : criu-3.18-4.module+el8.9.0+20326+387084d0.x86_64 6/31 2024-01-15T22:24:42.336 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : fuse-overlayfs-1.4.0-2.module+el8.7.0+16533+44634e 7/31 2024-01-15T22:24:42.336 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : fuse-overlayfs-1.12-1.module+el8.9.0+20326+387084d 8/31 2024-01-15T22:24:42.336 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libslirp-4.3.1-1.module+el8.7.0+16533+44634e24.x86 9/31 2024-01-15T22:24:42.336 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libslirp-4.4.0-1.module+el8.9.0+20326+387084d0.x86 10/31 2024-01-15T22:24:42.336 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : podman-3.0.1-15.module+el8.7.0+18021+1e286cc8.x86_ 11/31 2024-01-15T22:24:42.336 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : podman-3:4.6.1-4.module+el8.9.0+20326+387084d0.x86 12/31 2024-01-15T22:24:42.336 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : podman-docker-3.0.1-15.module+el8.7.0+18021+1e286c 13/31 2024-01-15T22:24:42.336 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : podman-docker-3:4.6.1-4.module+el8.9.0+20326+38708 14/31 2024-01-15T22:24:42.336 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : runc-1.0.0-73.rc95.module+el8.7.0+16533+44634e24.x 15/31 2024-01-15T22:24:42.336 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : runc-1:1.1.9-1.module+el8.9.0+20326+387084d0.x86_6 16/31 2024-01-15T22:24:42.336 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : container-selinux-2:2.189.0-1.module+el8.7.0+16533 17/31 2024-01-15T22:24:42.337 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : container-selinux-2:2.221.0-1.module+el8.9.0+20326 18/31 2024-01-15T22:24:42.337 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : containers-common-1:1.2.4-2.module+el8.7.0+16533+4 19/31 2024-01-15T22:24:42.337 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : containers-common-2:1-70.module+el8.9.0+20326+3870 20/31 2024-01-15T22:24:42.337 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : podman-catatonit-3.0.1-15.module+el8.7.0+18021+1e2 21/31 2024-01-15T22:24:42.337 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : podman-catatonit-3:4.6.1-4.module+el8.9.0+20326+38 22/31 2024-01-15T22:24:42.337 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : slirp4netns-1.1.8-1.module+el8.7.0+16533+44634e24. 23/31 2024-01-15T22:24:42.337 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : slirp4netns-1.2.1-1.module+el8.9.0+20326+387084d0. 24/31 2024-01-15T22:24:42.337 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : cockpit-podman-29-2.module+el8.7.0+16533+44634e24. 25/31 2024-01-15T22:24:42.337 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : crun-0.18-3.module+el8.7.0+16533+44634e24.x86_64 26/31 2024-01-15T22:24:42.337 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : skopeo-1:1.2.4-2.module+el8.7.0+16533+44634e24.x86 27/31 2024-01-15T22:24:42.338 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : toolbox-0.0.99.3-1.module+el8.7.0+16533+44634e24.x 28/31 2024-01-15T22:24:42.338 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : udica-0.2.4-1.module+el8.7.0+16533+44634e24.noarch 29/31 2024-01-15T22:24:42.338 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : buildah-1.19.9-6.module+el8.7.0+16533+44634e24.x86 30/31 2024-01-15T22:24:42.560 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: containernetworking-plugins-1:1.3.0-4.module+el8.9 31/31 2024-01-15T22:24:42.561 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : conmon-2:2.0.26-3.module+el8.7.0+16533+44634e24.x8 1/31 2024-01-15T22:24:42.561 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : conmon-3:2.1.8-1.module+el8.9.0+20326+387084d0.x86 2/31 2024-01-15T22:24:42.561 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : containernetworking-plugins-0.9.1-1.module+el8.7.0 3/31 2024-01-15T22:24:42.561 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : containernetworking-plugins-1:1.3.0-4.module+el8.9 4/31 2024-01-15T22:24:42.561 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : criu-3.15-1.module+el8.7.0+16533+44634e24.x86_64 5/31 2024-01-15T22:24:42.561 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : criu-3.18-4.module+el8.9.0+20326+387084d0.x86_64 6/31 2024-01-15T22:24:42.561 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : fuse-overlayfs-1.4.0-2.module+el8.7.0+16533+44634e 7/31 2024-01-15T22:24:42.561 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : fuse-overlayfs-1.12-1.module+el8.9.0+20326+387084d 8/31 2024-01-15T22:24:42.561 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libslirp-4.3.1-1.module+el8.7.0+16533+44634e24.x86 9/31 2024-01-15T22:24:42.561 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libslirp-4.4.0-1.module+el8.9.0+20326+387084d0.x86 10/31 2024-01-15T22:24:42.561 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : podman-3.0.1-15.module+el8.7.0+18021+1e286cc8.x86_ 11/31 2024-01-15T22:24:42.561 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : podman-3:4.6.1-4.module+el8.9.0+20326+387084d0.x86 12/31 2024-01-15T22:24:42.561 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : podman-docker-3.0.1-15.module+el8.7.0+18021+1e286c 13/31 2024-01-15T22:24:42.562 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : podman-docker-3:4.6.1-4.module+el8.9.0+20326+38708 14/31 2024-01-15T22:24:42.562 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : runc-1.0.0-73.rc95.module+el8.7.0+16533+44634e24.x 15/31 2024-01-15T22:24:42.562 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : runc-1:1.1.9-1.module+el8.9.0+20326+387084d0.x86_6 16/31 2024-01-15T22:24:42.562 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : container-selinux-2:2.189.0-1.module+el8.7.0+16533 17/31 2024-01-15T22:24:42.562 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : container-selinux-2:2.221.0-1.module+el8.9.0+20326 18/31 2024-01-15T22:24:42.562 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : containers-common-1:1.2.4-2.module+el8.7.0+16533+4 19/31 2024-01-15T22:24:42.562 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : containers-common-2:1-70.module+el8.9.0+20326+3870 20/31 2024-01-15T22:24:42.562 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : podman-catatonit-3.0.1-15.module+el8.7.0+18021+1e2 21/31 2024-01-15T22:24:42.562 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : podman-catatonit-3:4.6.1-4.module+el8.9.0+20326+38 22/31 2024-01-15T22:24:42.562 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : slirp4netns-1.1.8-1.module+el8.7.0+16533+44634e24. 23/31 2024-01-15T22:24:42.562 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : slirp4netns-1.2.1-1.module+el8.9.0+20326+387084d0. 24/31 2024-01-15T22:24:42.562 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : cockpit-podman-29-2.module+el8.7.0+16533+44634e24. 25/31 2024-01-15T22:24:42.562 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : crun-0.18-3.module+el8.7.0+16533+44634e24.x86_64 26/31 2024-01-15T22:24:42.562 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : skopeo-1:1.2.4-2.module+el8.7.0+16533+44634e24.x86 27/31 2024-01-15T22:24:42.562 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : toolbox-0.0.99.3-1.module+el8.7.0+16533+44634e24.x 28/31 2024-01-15T22:24:42.562 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : udica-0.2.4-1.module+el8.7.0+16533+44634e24.noarch 29/31 2024-01-15T22:24:42.563 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : buildah-1.19.9-6.module+el8.7.0+16533+44634e24.x86 30/31 2024-01-15T22:24:51.682 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 31/31 2024-01-15T22:24:51.683 INFO:teuthology.orchestra.run.smithi099.stdout:Installed products updated. 2024-01-15T22:24:52.220 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:24:52.221 INFO:teuthology.orchestra.run.smithi099.stdout:Downgraded: 2024-01-15T22:24:52.221 INFO:teuthology.orchestra.run.smithi099.stdout: conmon-2:2.0.26-3.module+el8.7.0+16533+44634e24.x86_64 2024-01-15T22:24:52.221 INFO:teuthology.orchestra.run.smithi099.stdout: container-selinux-2:2.189.0-1.module+el8.7.0+16533+44634e24.noarch 2024-01-15T22:24:52.221 INFO:teuthology.orchestra.run.smithi099.stdout: containernetworking-plugins-0.9.1-1.module+el8.7.0+16533+44634e24.x86_64 2024-01-15T22:24:52.221 INFO:teuthology.orchestra.run.smithi099.stdout: containers-common-1:1.2.4-2.module+el8.7.0+16533+44634e24.x86_64 2024-01-15T22:24:52.221 INFO:teuthology.orchestra.run.smithi099.stdout: criu-3.15-1.module+el8.7.0+16533+44634e24.x86_64 2024-01-15T22:24:52.221 INFO:teuthology.orchestra.run.smithi099.stdout: fuse-overlayfs-1.4.0-2.module+el8.7.0+16533+44634e24.x86_64 2024-01-15T22:24:52.221 INFO:teuthology.orchestra.run.smithi099.stdout: libslirp-4.3.1-1.module+el8.7.0+16533+44634e24.x86_64 2024-01-15T22:24:52.221 INFO:teuthology.orchestra.run.smithi099.stdout: podman-3.0.1-15.module+el8.7.0+18021+1e286cc8.x86_64 2024-01-15T22:24:52.221 INFO:teuthology.orchestra.run.smithi099.stdout: podman-catatonit-3.0.1-15.module+el8.7.0+18021+1e286cc8.x86_64 2024-01-15T22:24:52.221 INFO:teuthology.orchestra.run.smithi099.stdout: podman-docker-3.0.1-15.module+el8.7.0+18021+1e286cc8.noarch 2024-01-15T22:24:52.221 INFO:teuthology.orchestra.run.smithi099.stdout: runc-1.0.0-73.rc95.module+el8.7.0+16533+44634e24.x86_64 2024-01-15T22:24:52.221 INFO:teuthology.orchestra.run.smithi099.stdout: slirp4netns-1.1.8-1.module+el8.7.0+16533+44634e24.x86_64 2024-01-15T22:24:52.222 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-01-15T22:24:52.222 INFO:teuthology.orchestra.run.smithi099.stdout: buildah-1.19.9-6.module+el8.7.0+16533+44634e24.x86_64 2024-01-15T22:24:52.222 INFO:teuthology.orchestra.run.smithi099.stdout: cockpit-podman-29-2.module+el8.7.0+16533+44634e24.noarch 2024-01-15T22:24:52.222 INFO:teuthology.orchestra.run.smithi099.stdout: crun-0.18-3.module+el8.7.0+16533+44634e24.x86_64 2024-01-15T22:24:52.222 INFO:teuthology.orchestra.run.smithi099.stdout: skopeo-1:1.2.4-2.module+el8.7.0+16533+44634e24.x86_64 2024-01-15T22:24:52.222 INFO:teuthology.orchestra.run.smithi099.stdout: toolbox-0.0.99.3-1.module+el8.7.0+16533+44634e24.x86_64 2024-01-15T22:24:52.222 INFO:teuthology.orchestra.run.smithi099.stdout: udica-0.2.4-1.module+el8.7.0+16533+44634e24.noarch 2024-01-15T22:24:52.222 INFO:teuthology.orchestra.run.smithi099.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-15T22:24:52.222 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:24:52.222 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-01-15T22:24:52.391 DEBUG:teuthology.parallel:result is None 2024-01-15T22:24:55.004 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 31/31 2024-01-15T22:24:55.004 INFO:teuthology.orchestra.run.smithi162.stdout:Installed products updated. 2024-01-15T22:24:55.648 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-15T22:24:55.648 INFO:teuthology.orchestra.run.smithi162.stdout:Downgraded: 2024-01-15T22:24:55.648 INFO:teuthology.orchestra.run.smithi162.stdout: conmon-2:2.0.26-3.module+el8.7.0+16533+44634e24.x86_64 2024-01-15T22:24:55.648 INFO:teuthology.orchestra.run.smithi162.stdout: container-selinux-2:2.189.0-1.module+el8.7.0+16533+44634e24.noarch 2024-01-15T22:24:55.648 INFO:teuthology.orchestra.run.smithi162.stdout: containernetworking-plugins-0.9.1-1.module+el8.7.0+16533+44634e24.x86_64 2024-01-15T22:24:55.648 INFO:teuthology.orchestra.run.smithi162.stdout: containers-common-1:1.2.4-2.module+el8.7.0+16533+44634e24.x86_64 2024-01-15T22:24:55.648 INFO:teuthology.orchestra.run.smithi162.stdout: criu-3.15-1.module+el8.7.0+16533+44634e24.x86_64 2024-01-15T22:24:55.649 INFO:teuthology.orchestra.run.smithi162.stdout: fuse-overlayfs-1.4.0-2.module+el8.7.0+16533+44634e24.x86_64 2024-01-15T22:24:55.649 INFO:teuthology.orchestra.run.smithi162.stdout: libslirp-4.3.1-1.module+el8.7.0+16533+44634e24.x86_64 2024-01-15T22:24:55.649 INFO:teuthology.orchestra.run.smithi162.stdout: podman-3.0.1-15.module+el8.7.0+18021+1e286cc8.x86_64 2024-01-15T22:24:55.649 INFO:teuthology.orchestra.run.smithi162.stdout: podman-catatonit-3.0.1-15.module+el8.7.0+18021+1e286cc8.x86_64 2024-01-15T22:24:55.649 INFO:teuthology.orchestra.run.smithi162.stdout: podman-docker-3.0.1-15.module+el8.7.0+18021+1e286cc8.noarch 2024-01-15T22:24:55.649 INFO:teuthology.orchestra.run.smithi162.stdout: runc-1.0.0-73.rc95.module+el8.7.0+16533+44634e24.x86_64 2024-01-15T22:24:55.649 INFO:teuthology.orchestra.run.smithi162.stdout: slirp4netns-1.1.8-1.module+el8.7.0+16533+44634e24.x86_64 2024-01-15T22:24:55.649 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-01-15T22:24:55.649 INFO:teuthology.orchestra.run.smithi162.stdout: buildah-1.19.9-6.module+el8.7.0+16533+44634e24.x86_64 2024-01-15T22:24:55.649 INFO:teuthology.orchestra.run.smithi162.stdout: cockpit-podman-29-2.module+el8.7.0+16533+44634e24.noarch 2024-01-15T22:24:55.649 INFO:teuthology.orchestra.run.smithi162.stdout: crun-0.18-3.module+el8.7.0+16533+44634e24.x86_64 2024-01-15T22:24:55.649 INFO:teuthology.orchestra.run.smithi162.stdout: skopeo-1:1.2.4-2.module+el8.7.0+16533+44634e24.x86_64 2024-01-15T22:24:55.649 INFO:teuthology.orchestra.run.smithi162.stdout: toolbox-0.0.99.3-1.module+el8.7.0+16533+44634e24.x86_64 2024-01-15T22:24:55.649 INFO:teuthology.orchestra.run.smithi162.stdout: udica-0.2.4-1.module+el8.7.0+16533+44634e24.noarch 2024-01-15T22:24:55.649 INFO:teuthology.orchestra.run.smithi162.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-15T22:24:55.650 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-15T22:24:55.650 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-01-15T22:24:55.878 DEBUG:teuthology.parallel:result is None 2024-01-15T22:24:55.879 INFO:teuthology.run_tasks:Running task nvme_loop... 2024-01-15T22:24:55.894 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2024-01-15T22:24:55.894 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-01-15T22:24:55.894 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/scratch_devs of=/dev/stdout 2024-01-15T22:24:55.915 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-15T22:24:55.915 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_1 2024-01-15T22:24:55.978 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-15T22:24:55.978 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T22:24:55.978 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 175693 Links: 1 2024-01-15T22:24:55.978 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T22:24:55.978 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2024-01-15T22:24:55.978 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-01-15 22:24:42.032918632 +0000 2024-01-15T22:24:55.978 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-01-15 22:22:38.857040838 +0000 2024-01-15T22:24:55.978 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-01-15 22:22:38.857040838 +0000 2024-01-15T22:24:55.978 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2024-01-15T22:24:55.979 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-15T22:24:56.050 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-01-15T22:24:56.050 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-01-15T22:24:56.050 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000280989 s, 1.8 MB/s 2024-01-15T22:24:56.051 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-15T22:24:56.111 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_2 2024-01-15T22:24:56.171 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-15T22:24:56.171 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T22:24:56.171 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 172868 Links: 1 2024-01-15T22:24:56.171 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T22:24:56.171 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2024-01-15T22:24:56.171 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-01-15 22:24:42.032918632 +0000 2024-01-15T22:24:56.171 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-01-15 22:22:39.198033203 +0000 2024-01-15T22:24:56.171 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-01-15 22:22:39.198033203 +0000 2024-01-15T22:24:56.171 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2024-01-15T22:24:56.171 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-15T22:24:56.238 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-01-15T22:24:56.238 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-01-15T22:24:56.238 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000241688 s, 2.1 MB/s 2024-01-15T22:24:56.239 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-15T22:24:56.298 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_3 2024-01-15T22:24:56.356 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-15T22:24:56.356 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T22:24:56.357 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 178288 Links: 1 2024-01-15T22:24:56.357 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T22:24:56.357 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2024-01-15T22:24:56.357 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-01-15 22:24:42.033918610 +0000 2024-01-15T22:24:56.357 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-01-15 22:22:39.542025500 +0000 2024-01-15T22:24:56.357 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-01-15 22:22:39.542025500 +0000 2024-01-15T22:24:56.357 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2024-01-15T22:24:56.357 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-15T22:24:56.423 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-01-15T22:24:56.423 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-01-15T22:24:56.423 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000206647 s, 2.5 MB/s 2024-01-15T22:24:56.424 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-15T22:24:56.483 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_4 2024-01-15T22:24:56.540 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-15T22:24:56.540 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T22:24:56.540 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 178354 Links: 1 2024-01-15T22:24:56.540 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T22:24:56.540 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2024-01-15T22:24:56.540 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-01-15 22:24:42.033918610 +0000 2024-01-15T22:24:56.540 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-01-15 22:22:39.874018066 +0000 2024-01-15T22:24:56.540 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-01-15 22:22:39.874018066 +0000 2024-01-15T22:24:56.540 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2024-01-15T22:24:56.541 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-15T22:24:56.606 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-01-15T22:24:56.606 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-01-15T22:24:56.606 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000231981 s, 2.2 MB/s 2024-01-15T22:24:56.607 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-15T22:24:56.666 DEBUG:teuthology.orchestra.run.smithi099:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-01-15T22:24:56.783 INFO:teuthology.orchestra.run.smithi099.stdout:loop 2024-01-15T22:24:56.784 INFO:tasks.nvme_loop:Connecting nvme_loop smithi099:/dev/vg_nvme/lv_1... 2024-01-15T22:24:56.785 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-01-15T22:24:56.819 INFO:teuthology.orchestra.run.smithi099.stdout:1 2024-01-15T22:24:56.854 INFO:teuthology.orchestra.run.smithi099.stdout:/dev/vg_nvme/lv_11 2024-01-15T22:24:56.894 INFO:tasks.nvme_loop:Connecting nvme_loop smithi099:/dev/vg_nvme/lv_2... 2024-01-15T22:24:56.895 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-01-15T22:24:56.971 INFO:teuthology.orchestra.run.smithi099.stdout:1 2024-01-15T22:24:57.007 INFO:teuthology.orchestra.run.smithi099.stdout:/dev/vg_nvme/lv_21 2024-01-15T22:24:57.046 INFO:tasks.nvme_loop:Connecting nvme_loop smithi099:/dev/vg_nvme/lv_3... 2024-01-15T22:24:57.047 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-01-15T22:24:57.122 INFO:teuthology.orchestra.run.smithi099.stdout:1 2024-01-15T22:24:57.157 INFO:teuthology.orchestra.run.smithi099.stdout:/dev/vg_nvme/lv_31 2024-01-15T22:24:57.196 INFO:tasks.nvme_loop:Connecting nvme_loop smithi099:/dev/vg_nvme/lv_4... 2024-01-15T22:24:57.197 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-01-15T22:24:57.272 INFO:teuthology.orchestra.run.smithi099.stdout:1 2024-01-15T22:24:57.307 INFO:teuthology.orchestra.run.smithi099.stdout:/dev/vg_nvme/lv_41 2024-01-15T22:24:57.348 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-01-15T22:24:57.348 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/scratch_devs of=/dev/stdout 2024-01-15T22:24:57.406 DEBUG:teuthology.orchestra.run.smithi099:> sudo nvme list 2024-01-15T22:24:57.472 INFO:teuthology.orchestra.run.smithi099.stdout:Node SN Model Namespace Usage Format FW Rev 2024-01-15T22:24:57.472 INFO:teuthology.orchestra.run.smithi099.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-01-15T22:24:57.472 INFO:teuthology.orchestra.run.smithi099.stdout:/dev/nvme0n1 PHFT62040101400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-01-15T22:24:57.472 INFO:teuthology.orchestra.run.smithi099.stdout:/dev/nvme1n1 414879ea7c63614a2fa7 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2024-01-15T22:24:57.472 INFO:teuthology.orchestra.run.smithi099.stdout:/dev/nvme2n1 a17e7443f59d57d4fae8 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2024-01-15T22:24:57.472 INFO:teuthology.orchestra.run.smithi099.stdout:/dev/nvme3n1 f3546ab840a2237fbb4a Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2024-01-15T22:24:57.472 INFO:teuthology.orchestra.run.smithi099.stdout:/dev/nvme4n1 999432559d2e4551a58a Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2024-01-15T22:24:57.473 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-01-15T22:24:57.473 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-01-15T22:24:57.473 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/scratch_devs 2024-01-15T22:24:57.540 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-01-15T22:24:57.540 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/scratch_devs of=/dev/stdout 2024-01-15T22:24:57.562 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-15T22:24:57.562 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_1 2024-01-15T22:24:57.622 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-15T22:24:57.622 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T22:24:57.623 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 6h/6d Inode: 171491 Links: 1 2024-01-15T22:24:57.623 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T22:24:57.623 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-01-15T22:24:57.623 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-01-15 22:24:42.272935621 +0000 2024-01-15T22:24:57.623 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-01-15 22:22:58.909242926 +0000 2024-01-15T22:24:57.623 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-01-15 22:22:58.909242926 +0000 2024-01-15T22:24:57.623 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: - 2024-01-15T22:24:57.623 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-15T22:24:57.694 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-01-15T22:24:57.694 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-01-15T22:24:57.694 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000312305 s, 1.6 MB/s 2024-01-15T22:24:57.696 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-15T22:24:57.756 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_2 2024-01-15T22:24:57.815 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-15T22:24:57.815 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T22:24:57.815 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 6h/6d Inode: 176417 Links: 1 2024-01-15T22:24:57.815 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T22:24:57.815 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-01-15T22:24:57.815 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-01-15 22:24:42.272935621 +0000 2024-01-15T22:24:57.816 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-01-15 22:22:59.256236066 +0000 2024-01-15T22:24:57.816 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-01-15 22:22:59.256236066 +0000 2024-01-15T22:24:57.816 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: - 2024-01-15T22:24:57.816 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-15T22:24:57.883 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-01-15T22:24:57.884 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-01-15T22:24:57.884 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.0002015 s, 2.5 MB/s 2024-01-15T22:24:57.885 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-15T22:24:57.944 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_3 2024-01-15T22:24:58.004 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-15T22:24:58.005 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T22:24:58.005 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 6h/6d Inode: 178262 Links: 1 2024-01-15T22:24:58.005 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T22:24:58.005 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-01-15T22:24:58.005 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-01-15 22:24:42.272935621 +0000 2024-01-15T22:24:58.005 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-01-15 22:22:59.588229503 +0000 2024-01-15T22:24:58.005 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-01-15 22:22:59.588229503 +0000 2024-01-15T22:24:58.005 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: - 2024-01-15T22:24:58.005 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-15T22:24:58.073 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-01-15T22:24:58.073 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-01-15T22:24:58.073 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000269993 s, 1.9 MB/s 2024-01-15T22:24:58.075 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-15T22:24:58.134 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_4 2024-01-15T22:24:58.193 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-15T22:24:58.193 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T22:24:58.193 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 6h/6d Inode: 171680 Links: 1 2024-01-15T22:24:58.194 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T22:24:58.194 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-01-15T22:24:58.194 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-01-15 22:24:42.272935621 +0000 2024-01-15T22:24:58.194 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-01-15 22:22:59.920222939 +0000 2024-01-15T22:24:58.194 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-01-15 22:22:59.920222939 +0000 2024-01-15T22:24:58.194 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: - 2024-01-15T22:24:58.194 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-15T22:24:58.261 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-01-15T22:24:58.261 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-01-15T22:24:58.261 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000240447 s, 2.1 MB/s 2024-01-15T22:24:58.262 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-15T22:24:58.321 DEBUG:teuthology.orchestra.run.smithi162:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-01-15T22:24:58.443 INFO:teuthology.orchestra.run.smithi162.stdout:loop 2024-01-15T22:24:58.444 INFO:tasks.nvme_loop:Connecting nvme_loop smithi162:/dev/vg_nvme/lv_1... 2024-01-15T22:24:58.444 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-01-15T22:24:58.479 INFO:teuthology.orchestra.run.smithi162.stdout:1 2024-01-15T22:24:58.515 INFO:teuthology.orchestra.run.smithi162.stdout:/dev/vg_nvme/lv_11 2024-01-15T22:24:58.555 INFO:tasks.nvme_loop:Connecting nvme_loop smithi162:/dev/vg_nvme/lv_2... 2024-01-15T22:24:58.555 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-01-15T22:24:58.632 INFO:teuthology.orchestra.run.smithi162.stdout:1 2024-01-15T22:24:58.668 INFO:teuthology.orchestra.run.smithi162.stdout:/dev/vg_nvme/lv_21 2024-01-15T22:24:58.707 INFO:tasks.nvme_loop:Connecting nvme_loop smithi162:/dev/vg_nvme/lv_3... 2024-01-15T22:24:58.707 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-01-15T22:24:58.784 INFO:teuthology.orchestra.run.smithi162.stdout:1 2024-01-15T22:24:58.819 INFO:teuthology.orchestra.run.smithi162.stdout:/dev/vg_nvme/lv_31 2024-01-15T22:24:58.857 INFO:tasks.nvme_loop:Connecting nvme_loop smithi162:/dev/vg_nvme/lv_4... 2024-01-15T22:24:58.857 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-01-15T22:24:58.934 INFO:teuthology.orchestra.run.smithi162.stdout:1 2024-01-15T22:24:58.969 INFO:teuthology.orchestra.run.smithi162.stdout:/dev/vg_nvme/lv_41 2024-01-15T22:24:59.008 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-01-15T22:24:59.009 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/scratch_devs of=/dev/stdout 2024-01-15T22:24:59.066 DEBUG:teuthology.orchestra.run.smithi162:> sudo nvme list 2024-01-15T22:24:59.131 INFO:teuthology.orchestra.run.smithi162.stdout:Node SN Model Namespace Usage Format FW Rev 2024-01-15T22:24:59.131 INFO:teuthology.orchestra.run.smithi162.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-01-15T22:24:59.131 INFO:teuthology.orchestra.run.smithi162.stdout:/dev/nvme0n1 CVFT623300JU400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-01-15T22:24:59.131 INFO:teuthology.orchestra.run.smithi162.stdout:/dev/nvme1n1 c11057747f48018dea28 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2024-01-15T22:24:59.132 INFO:teuthology.orchestra.run.smithi162.stdout:/dev/nvme2n1 7e0d19d2f2e4d7c3afd4 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2024-01-15T22:24:59.132 INFO:teuthology.orchestra.run.smithi162.stdout:/dev/nvme3n1 fc758f8a40a93ea0dd67 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2024-01-15T22:24:59.132 INFO:teuthology.orchestra.run.smithi162.stdout:/dev/nvme4n1 885a458187946e964758 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2024-01-15T22:24:59.132 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-01-15T22:24:59.133 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-01-15T22:24:59.133 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/scratch_devs 2024-01-15T22:24:59.199 INFO:teuthology.run_tasks:Running task cephadm... 2024-01-15T22:24:59.264 INFO:tasks.cephadm:Config: {'roleless': True, 'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '445562ab4bc3ddfb386936119050695810860bcb'} 2024-01-15T22:24:59.264 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb 2024-01-15T22:24:59.265 INFO:tasks.cephadm:Cluster fsid is eb17a8d4-b3f4-11ee-95ad-87774f69a715 2024-01-15T22:24:59.265 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-01-15T22:24:59.265 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-01-15T22:24:59.265 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi099': '172.21.15.99', 'mon.smithi162': '172.21.15.162'} 2024-01-15T22:24:59.265 INFO:tasks.cephadm:Normalizing hostnames... 2024-01-15T22:24:59.265 DEBUG:teuthology.orchestra.run.smithi099:> sudo hostname $(hostname -s) 2024-01-15T22:24:59.298 DEBUG:teuthology.orchestra.run.smithi162:> sudo hostname $(hostname -s) 2024-01-15T22:24:59.325 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph-ci.git ref 445562ab4bc3ddfb386936119050695810860bcb)... 2024-01-15T22:24:59.326 DEBUG:teuthology.orchestra.run.smithi099:> curl --silent https://raw.githubusercontent.com/ceph/ceph-ci/445562ab4bc3ddfb386936119050695810860bcb/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-15T22:24:59.888 INFO:teuthology.orchestra.run.smithi099.stdout:-rw-rw-r--. 1 ubuntu ubuntu 355729 Jan 15 22:24 /home/ubuntu/cephtest/cephadm 2024-01-15T22:24:59.888 DEBUG:teuthology.orchestra.run.smithi162:> curl --silent https://raw.githubusercontent.com/ceph/ceph-ci/445562ab4bc3ddfb386936119050695810860bcb/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-15T22:25:01.522 INFO:teuthology.orchestra.run.smithi162.stdout:-rw-rw-r--. 1 ubuntu ubuntu 355729 Jan 15 22:25 /home/ubuntu/cephtest/cephadm 2024-01-15T22:25:01.522 DEBUG:teuthology.orchestra.run.smithi099:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-15T22:25:01.544 DEBUG:teuthology.orchestra.run.smithi162:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-15T22:25:01.572 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /etc/ceph 2024-01-15T22:25:01.617 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /etc/ceph 2024-01-15T22:25:01.648 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 777 /etc/ceph 2024-01-15T22:25:01.688 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 777 /etc/ceph 2024-01-15T22:25:01.717 INFO:tasks.cephadm:Writing seed config... 2024-01-15T22:25:01.718 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-01-15T22:25:01.718 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-01-15T22:25:01.718 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-01-15T22:25:01.718 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-01-15T22:25:01.718 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-01-15T22:25:01.718 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-01-15T22:25:01.718 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-01-15T22:25:01.718 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-01-15T22:25:01.719 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-01-15T22:25:01.719 DEBUG:teuthology.orchestra.run.smithi099:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-01-15T22:25:01.747 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = eb17a8d4-b3f4-11ee-95ad-87774f69a715 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-01-15T22:25:01.747 DEBUG:teuthology.orchestra.run.smithi099:mon.smithi099> sudo journalctl -f -n 0 -u ceph-eb17a8d4-b3f4-11ee-95ad-87774f69a715@mon.smithi099.service 2024-01-15T22:25:01.788 INFO:tasks.cephadm:Bootstrapping... 2024-01-15T22:25:01.789 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb -v bootstrap --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.99 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-01-15T22:25:01.817 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:-- Logs begin at Mon 2024-01-15 22:13:12 UTC. -- 2024-01-15T22:25:02.074 INFO:teuthology.orchestra.run.smithi099.stderr:-------------------------------------------------------------------------------- 2024-01-15T22:25:02.074 INFO:teuthology.orchestra.run.smithi099.stderr:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb', '-v', 'bootstrap', '--fsid', 'eb17a8d4-b3f4-11ee-95ad-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.99', '--skip-admin-label'] 2024-01-15T22:25:03.002 INFO:teuthology.orchestra.run.smithi099.stderr:/bin/podman: 3.0.1 2024-01-15T22:25:03.020 INFO:teuthology.orchestra.run.smithi099.stderr:Verifying podman|docker is present... 2024-01-15T22:25:03.063 INFO:teuthology.orchestra.run.smithi099.stderr:/bin/podman: 3.0.1 2024-01-15T22:25:03.082 INFO:teuthology.orchestra.run.smithi099.stderr:Verifying lvm2 is present... 2024-01-15T22:25:03.082 INFO:teuthology.orchestra.run.smithi099.stderr:Verifying time synchronization is in place... 2024-01-15T22:25:03.089 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-01-15T22:25:03.095 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: inactive 2024-01-15T22:25:03.101 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: enabled 2024-01-15T22:25:03.107 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: active 2024-01-15T22:25:03.107 INFO:teuthology.orchestra.run.smithi099.stderr:Unit chronyd.service is enabled and running 2024-01-15T22:25:03.107 INFO:teuthology.orchestra.run.smithi099.stderr:Repeating the final host check... 2024-01-15T22:25:03.145 INFO:teuthology.orchestra.run.smithi099.stderr:/bin/podman: 3.0.1 2024-01-15T22:25:03.164 INFO:teuthology.orchestra.run.smithi099.stderr:podman (/bin/podman) version 3.0.1 is present 2024-01-15T22:25:03.164 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl is present 2024-01-15T22:25:03.164 INFO:teuthology.orchestra.run.smithi099.stderr:lvcreate is present 2024-01-15T22:25:03.170 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-01-15T22:25:03.175 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: inactive 2024-01-15T22:25:03.182 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: enabled 2024-01-15T22:25:03.187 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: active 2024-01-15T22:25:03.188 INFO:teuthology.orchestra.run.smithi099.stderr:Unit chronyd.service is enabled and running 2024-01-15T22:25:03.188 INFO:teuthology.orchestra.run.smithi099.stderr:Host looks OK 2024-01-15T22:25:03.188 INFO:teuthology.orchestra.run.smithi099.stderr:Cluster fsid: eb17a8d4-b3f4-11ee-95ad-87774f69a715 2024-01-15T22:25:03.188 INFO:teuthology.orchestra.run.smithi099.stderr:Acquiring lock 139773537378432 on /run/cephadm/eb17a8d4-b3f4-11ee-95ad-87774f69a715.lock 2024-01-15T22:25:03.188 INFO:teuthology.orchestra.run.smithi099.stderr:Lock 139773537378432 acquired on /run/cephadm/eb17a8d4-b3f4-11ee-95ad-87774f69a715.lock 2024-01-15T22:25:03.189 INFO:teuthology.orchestra.run.smithi099.stderr:Verifying IP 172.21.15.99 port 3300 ... 2024-01-15T22:25:03.189 INFO:teuthology.orchestra.run.smithi099.stderr:Verifying IP 172.21.15.99 port 6789 ... 2024-01-15T22:25:03.189 INFO:teuthology.orchestra.run.smithi099.stderr:Base mon IP(s) is [172.21.15.99:3300, 172.21.15.99:6789], mon addrv is [v2:172.21.15.99:3300,v1:172.21.15.99:6789] 2024-01-15T22:25:03.192 INFO:teuthology.orchestra.run.smithi099.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp metric 100 2024-01-15T22:25:03.192 INFO:teuthology.orchestra.run.smithi099.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.99 metric 100 2024-01-15T22:25:03.194 INFO:teuthology.orchestra.run.smithi099.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-01-15T22:25:03.194 INFO:teuthology.orchestra.run.smithi099.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-01-15T22:25:03.195 INFO:teuthology.orchestra.run.smithi099.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-01-15T22:25:03.197 INFO:teuthology.orchestra.run.smithi099.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-01-15T22:25:03.197 INFO:teuthology.orchestra.run.smithi099.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-01-15T22:25:03.197 INFO:teuthology.orchestra.run.smithi099.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-01-15T22:25:03.197 INFO:teuthology.orchestra.run.smithi099.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-01-15T22:25:03.197 INFO:teuthology.orchestra.run.smithi099.stderr:/sbin/ip: inet6 fe80::1c11:9d47:b034:e822/64 scope link noprefixroute 2024-01-15T22:25:03.197 INFO:teuthology.orchestra.run.smithi099.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-01-15T22:25:03.198 INFO:teuthology.orchestra.run.smithi099.stderr:Mon IP `172.21.15.99` is in CIDR network `172.21.0.0/20` 2024-01-15T22:25:03.198 INFO:teuthology.orchestra.run.smithi099.stderr:Mon IP `172.21.15.99` is in CIDR network `172.21.0.0/20` 2024-01-15T22:25:03.198 INFO:teuthology.orchestra.run.smithi099.stderr:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-01-15T22:25:03.199 INFO:teuthology.orchestra.run.smithi099.stderr:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-01-15T22:25:03.199 INFO:teuthology.orchestra.run.smithi099.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb... 2024-01-15T22:25:03.266 INFO:teuthology.orchestra.run.smithi099.stderr:/bin/podman: Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb... 2024-01-15T22:25:03.371 INFO:teuthology.orchestra.run.smithi099.stderr:/bin/podman: Getting image source signatures 2024-01-15T22:25:03.485 INFO:teuthology.orchestra.run.smithi099.stderr:/bin/podman: Copying blob sha256:6c507150f7e9f028ef978d633e702dab944bbda82bfbc46f6307495cf6dd9fc7 2024-01-15T22:25:03.543 INFO:teuthology.orchestra.run.smithi099.stderr:/bin/podman: Copying blob sha256:9bc82cfe809169b39ceaa8362ba30ca8525e0ac28e35f12009a7a395fd81b5b0 2024-01-15T22:25:25.206 INFO:teuthology.orchestra.run.smithi099.stderr:/bin/podman: Copying config sha256:163fd1944c31b631006e5d238574cf3311ed176b42b55b0b64e2387958260fc0 2024-01-15T22:25:25.272 INFO:teuthology.orchestra.run.smithi099.stderr:/bin/podman: Writing manifest to image destination 2024-01-15T22:25:25.272 INFO:teuthology.orchestra.run.smithi099.stderr:/bin/podman: Storing signatures 2024-01-15T22:25:37.378 INFO:teuthology.orchestra.run.smithi099.stderr:/bin/podman: 163fd1944c31b631006e5d238574cf3311ed176b42b55b0b64e2387958260fc0 2024-01-15T22:25:37.915 INFO:teuthology.orchestra.run.smithi099.stderr:ceph: ceph version 16.2.14-456-g445562ab (445562ab4bc3ddfb386936119050695810860bcb) pacific (stable) 2024-01-15T22:25:49.213 INFO:teuthology.orchestra.run.smithi099.stderr:Ceph version: ceph version 16.2.14-456-g445562ab (445562ab4bc3ddfb386936119050695810860bcb) pacific (stable) 2024-01-15T22:25:49.213 INFO:teuthology.orchestra.run.smithi099.stderr:Extracting ceph user uid/gid from container image... 2024-01-15T22:25:49.583 INFO:teuthology.orchestra.run.smithi099.stderr:stat: 167 167 2024-01-15T22:25:50.058 INFO:teuthology.orchestra.run.smithi099.stderr:Creating initial keys... 2024-01-15T22:25:50.416 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-authtool: AQDusKVlIX2bFxAAZaZZEF3B4ImRxY0r6UdkaA== 2024-01-15T22:25:51.249 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-authtool: AQDvsKVloKGjDRAAHP3TigU11hnwxauHEaKHQA== 2024-01-15T22:25:52.048 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-authtool: AQDwsKVlm6e2ARAA4BTas/C35yM5mCN9VOmjUQ== 2024-01-15T22:25:52.543 INFO:teuthology.orchestra.run.smithi099.stderr:Creating initial monmap... 2024-01-15T22:25:52.907 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-15T22:25:52.908 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to eb17a8d4-b3f4-11ee-95ad-87774f69a715 2024-01-15T22:25:52.908 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-15T22:25:53.359 INFO:teuthology.orchestra.run.smithi099.stderr:monmaptool for smithi099 [v2:172.21.15.99:3300,v1:172.21.15.99:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-15T22:25:53.359 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/monmaptool: set fsid to eb17a8d4-b3f4-11ee-95ad-87774f69a715 2024-01-15T22:25:53.359 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-15T22:25:53.359 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-01-15T22:25:53.359 INFO:teuthology.orchestra.run.smithi099.stderr:Creating mon... 2024-01-15T22:25:53.732 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.694+0000 7f56197bc880 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-15T22:25:53.732 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.694+0000 7f56197bc880 1 imported monmap: 2024-01-15T22:25:53.732 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: epoch 0 2024-01-15T22:25:53.732 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 2024-01-15T22:25:53.732 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: last_changed 2024-01-15T22:25:52.892888+0000 2024-01-15T22:25:53.732 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: created 2024-01-15T22:25:52.892888+0000 2024-01-15T22:25:53.732 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-01-15T22:25:53.732 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-01-15T22:25:53.732 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.smithi099 2024-01-15T22:25:53.733 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: 2024-01-15T22:25:53.733 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.694+0000 7f56197bc880 0 /usr/bin/ceph-mon: set fsid to eb17a8d4-b3f4-11ee-95ad-87774f69a715 2024-01-15T22:25:53.733 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: RocksDB version: 6.8.1 2024-01-15T22:25:53.733 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: 2024-01-15T22:25:53.733 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-15T22:25:53.733 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Compile date Jan 12 2024 2024-01-15T22:25:53.733 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: DB SUMMARY 2024-01-15T22:25:53.733 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: 2024-01-15T22:25:53.733 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi099/store.db dir, Total Num: 0, files: 2024-01-15T22:25:53.733 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: 2024-01-15T22:25:53.733 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi099/store.db: 2024-01-15T22:25:53.733 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: 2024-01-15T22:25:53.733 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.error_if_exists: 0 2024-01-15T22:25:53.733 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.create_if_missing: 1 2024-01-15T22:25:53.733 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.paranoid_checks: 1 2024-01-15T22:25:53.734 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.env: 0x55e0b6d15080 2024-01-15T22:25:53.734 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.fs: Posix File System 2024-01-15T22:25:53.734 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.info_log: 0x55e0b8437380 2024-01-15T22:25:53.734 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-15T22:25:53.734 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.statistics: (nil) 2024-01-15T22:25:53.734 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.use_fsync: 0 2024-01-15T22:25:53.734 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.max_log_file_size: 0 2024-01-15T22:25:53.734 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-15T22:25:53.734 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-15T22:25:53.734 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-15T22:25:53.734 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-15T22:25:53.734 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.allow_fallocate: 1 2024-01-15T22:25:53.734 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-15T22:25:53.734 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-15T22:25:53.734 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.use_direct_reads: 0 2024-01-15T22:25:53.735 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-15T22:25:53.735 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.create_missing_column_families: 0 2024-01-15T22:25:53.735 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.db_log_dir: 2024-01-15T22:25:53.735 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi099/store.db 2024-01-15T22:25:53.735 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-15T22:25:53.735 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.max_subcompactions: 1 2024-01-15T22:25:53.735 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.max_background_flushes: -1 2024-01-15T22:25:53.735 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-15T22:25:53.735 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-15T22:25:53.735 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-15T22:25:53.735 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-15T22:25:53.735 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-15T22:25:53.735 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.advise_random_on_open: 1 2024-01-15T22:25:53.735 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-15T22:25:53.735 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.write_buffer_manager: 0x55e0b843d980 2024-01-15T22:25:53.735 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-15T22:25:53.736 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-15T22:25:53.736 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-15T22:25:53.736 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-15T22:25:53.736 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.rate_limiter: (nil) 2024-01-15T22:25:53.736 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-15T22:25:53.736 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-15T22:25:53.736 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-15T22:25:53.736 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-15T22:25:53.736 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.unordered_write: 0 2024-01-15T22:25:53.736 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-15T22:25:53.736 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-15T22:25:53.736 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-15T22:25:53.736 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-15T22:25:53.736 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.row_cache: None 2024-01-15T22:25:53.736 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.wal_filter: None 2024-01-15T22:25:53.736 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-15T22:25:53.737 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-15T22:25:53.737 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.preserve_deletes: 0 2024-01-15T22:25:53.737 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.two_write_queues: 0 2024-01-15T22:25:53.737 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.manual_wal_flush: 0 2024-01-15T22:25:53.737 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.atomic_flush: 0 2024-01-15T22:25:53.737 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-15T22:25:53.737 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-15T22:25:53.737 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-15T22:25:53.737 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.log_readahead_size: 0 2024-01-15T22:25:53.737 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-15T22:25:53.737 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.max_background_jobs: 2 2024-01-15T22:25:53.737 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.max_background_compactions: -1 2024-01-15T22:25:53.737 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-15T22:25:53.737 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-15T22:25:53.737 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-15T22:25:53.738 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.max_total_wal_size: 0 2024-01-15T22:25:53.738 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-15T22:25:53.738 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-15T22:25:53.738 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-15T22:25:53.738 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-15T22:25:53.738 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.max_open_files: -1 2024-01-15T22:25:53.738 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.bytes_per_sync: 0 2024-01-15T22:25:53.738 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-15T22:25:53.738 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-15T22:25:53.738 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-15T22:25:53.738 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Compression algorithms supported: 2024-01-15T22:25:53.738 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-15T22:25:53.738 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: kZSTD supported: 0 2024-01-15T22:25:53.738 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: kXpressCompression supported: 0 2024-01-15T22:25:53.738 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-15T22:25:53.738 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: kLZ4Compression supported: 1 2024-01-15T22:25:53.739 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: kBZip2Compression supported: 0 2024-01-15T22:25:53.739 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: kZlibCompression supported: 1 2024-01-15T22:25:53.739 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: kSnappyCompression supported: 1 2024-01-15T22:25:53.739 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.695+0000 7f56197bc880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-15T22:25:53.739 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-01-15T22:25:53.739 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: 2024-01-15T22:25:53.739 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi099/store.db/MANIFEST-000001 2024-01-15T22:25:53.739 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: 2024-01-15T22:25:53.739 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-15T22:25:53.739 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: 2024-01-15T22:25:53.739 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-15T22:25:53.739 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.merge_operator: 2024-01-15T22:25:53.739 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.compaction_filter: None 2024-01-15T22:25:53.739 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.compaction_filter_factory: None 2024-01-15T22:25:53.739 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-15T22:25:53.739 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-15T22:25:53.740 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e0b8347d30) 2024-01-15T22:25:53.740 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-01-15T22:25:53.740 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-15T22:25:53.740 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-15T22:25:53.740 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-01-15T22:25:53.740 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: index_type: 0 2024-01-15T22:25:53.740 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-01-15T22:25:53.740 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-01-15T22:25:53.740 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-01-15T22:25:53.740 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-01-15T22:25:53.740 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: checksum: 1 2024-01-15T22:25:53.740 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-01-15T22:25:53.740 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: block_cache: 0x55e0b8380d10 2024-01-15T22:25:53.740 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-01-15T22:25:53.740 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-01-15T22:25:53.740 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-01-15T22:25:53.741 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-01-15T22:25:53.741 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-01-15T22:25:53.741 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-01-15T22:25:53.741 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-01-15T22:25:53.741 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-01-15T22:25:53.741 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-01-15T22:25:53.741 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-01-15T22:25:53.741 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-01-15T22:25:53.741 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-01-15T22:25:53.741 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-01-15T22:25:53.741 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-01-15T22:25:53.741 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-01-15T22:25:53.741 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-15T22:25:53.741 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-01-15T22:25:53.741 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-01-15T22:25:53.742 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-01-15T22:25:53.742 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: format_version: 2 2024-01-15T22:25:53.742 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-01-15T22:25:53.742 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: block_align: 0 2024-01-15T22:25:53.742 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: 2024-01-15T22:25:53.742 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-15T22:25:53.742 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-15T22:25:53.742 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.compression: NoCompression 2024-01-15T22:25:53.742 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-15T22:25:53.742 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-15T22:25:53.742 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-15T22:25:53.742 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.num_levels: 7 2024-01-15T22:25:53.742 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-15T22:25:53.742 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-15T22:25:53.742 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-15T22:25:53.742 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-15T22:25:53.743 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-15T22:25:53.743 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-15T22:25:53.743 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-15T22:25:53.743 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-15T22:25:53.743 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-15T22:25:53.743 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-15T22:25:53.743 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.compression_opts.level: 32767 2024-01-15T22:25:53.743 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-15T22:25:53.743 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-15T22:25:53.743 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-15T22:25:53.743 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.compression_opts.enabled: false 2024-01-15T22:25:53.743 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-15T22:25:53.743 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-15T22:25:53.743 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-15T22:25:53.743 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-15T22:25:53.744 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-15T22:25:53.744 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-15T22:25:53.744 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-15T22:25:53.744 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-15T22:25:53.744 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-15T22:25:53.744 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-15T22:25:53.744 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-15T22:25:53.744 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-15T22:25:53.744 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-15T22:25:53.744 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-15T22:25:53.744 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-15T22:25:53.744 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-15T22:25:53.744 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-15T22:25:53.744 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.arena_block_size: 4194304 2024-01-15T22:25:53.744 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-15T22:25:53.745 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-15T22:25:53.745 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-15T22:25:53.745 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-15T22:25:53.745 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-15T22:25:53.745 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-15T22:25:53.745 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-15T22:25:53.745 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-15T22:25:53.745 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-15T22:25:53.745 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-15T22:25:53.745 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-15T22:25:53.745 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-15T22:25:53.745 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-15T22:25:53.745 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-15T22:25:53.745 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.table_properties_collectors: 2024-01-15T22:25:53.745 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.inplace_update_support: 0 2024-01-15T22:25:53.745 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-15T22:25:53.745 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-15T22:25:53.746 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-15T22:25:53.746 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-15T22:25:53.746 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.bloom_locality: 0 2024-01-15T22:25:53.746 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.max_successive_merges: 0 2024-01-15T22:25:53.746 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-15T22:25:53.746 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-15T22:25:53.746 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.force_consistency_checks: 0 2024-01-15T22:25:53.746 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-15T22:25:53.746 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.ttl: 2592000 2024-01-15T22:25:53.746 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.696+0000 7f56197bc880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-15T22:25:53.746 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.697+0000 7f56197bc880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi099/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-15T22:25:53.746 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: 2024-01-15T22:25:53.746 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.697+0000 7f56197bc880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-01-15T22:25:53.746 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: 2024-01-15T22:25:53.746 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.697+0000 7f56197bc880 4 rocksdb: DB pointer 0x55e0b844b800 2024-01-15T22:25:53.746 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.697+0000 7f5602631700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-15T22:25:53.747 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.697+0000 7f5602631700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-15T22:25:53.747 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-01-15T22:25:53.747 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-15T22:25:53.747 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-15T22:25:53.747 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-15T22:25:53.747 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-15T22:25:53.747 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-15T22:25:53.747 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-15T22:25:53.747 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-15T22:25:53.747 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: 2024-01-15T22:25:53.747 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-15T22:25:53.747 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-15T22:25:53.747 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-15T22:25:53.747 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-15T22:25:53.747 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-15T22:25:53.747 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: 2024-01-15T22:25:53.748 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-15T22:25:53.748 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-15T22:25:53.748 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-15T22:25:53.748 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-15T22:25:53.748 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-01-15T22:25:53.748 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-15T22:25:53.748 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-01-15T22:25:53.748 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-01-15T22:25:53.748 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-01-15T22:25:53.748 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-15T22:25:53.748 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-15T22:25:53.748 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-15T22:25:53.748 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: 2024-01-15T22:25:53.748 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-01-15T22:25:53.748 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: 2024-01-15T22:25:53.749 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-15T22:25:53.749 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-15T22:25:53.749 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-15T22:25:53.749 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-15T22:25:53.749 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-15T22:25:53.749 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: 2024-01-15T22:25:53.749 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-15T22:25:53.749 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-15T22:25:53.749 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-15T22:25:53.749 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-15T22:25:53.749 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-01-15T22:25:53.749 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-15T22:25:53.749 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-01-15T22:25:53.749 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-01-15T22:25:53.749 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-01-15T22:25:53.750 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-15T22:25:53.750 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-15T22:25:53.750 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-15T22:25:53.750 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: 2024-01-15T22:25:53.750 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-01-15T22:25:53.750 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: 2024-01-15T22:25:53.750 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.699+0000 7f56197bc880 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-01-15T22:25:53.750 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.699+0000 7f56197bc880 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-01-15T22:25:53.750 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-15T22:25:53.699+0000 7f56197bc880 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi099 for mon.smithi099 2024-01-15T22:25:54.199 INFO:teuthology.orchestra.run.smithi099.stderr:create mon.smithi099 on 2024-01-15T22:25:54.309 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-01-15T22:25:54.405 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-eb17a8d4-b3f4-11ee-95ad-87774f69a715.target → /etc/systemd/system/ceph-eb17a8d4-b3f4-11ee-95ad-87774f69a715.target. 2024-01-15T22:25:54.406 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-eb17a8d4-b3f4-11ee-95ad-87774f69a715.target → /etc/systemd/system/ceph-eb17a8d4-b3f4-11ee-95ad-87774f69a715.target. 2024-01-15T22:25:54.617 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: Failed to reset failed state of unit ceph-eb17a8d4-b3f4-11ee-95ad-87774f69a715@mon.smithi099.service: Unit ceph-eb17a8d4-b3f4-11ee-95ad-87774f69a715@mon.smithi099.service not loaded. 2024-01-15T22:25:54.623 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: Created symlink /etc/systemd/system/ceph-eb17a8d4-b3f4-11ee-95ad-87774f69a715.target.wants/ceph-eb17a8d4-b3f4-11ee-95ad-87774f69a715@mon.smithi099.service → /etc/systemd/system/ceph-eb17a8d4-b3f4-11ee-95ad-87774f69a715@.service. 2024-01-15T22:25:54.903 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:54 smithi099 systemd[1]: Starting Ceph mon.smithi099 for eb17a8d4-b3f4-11ee-95ad-87774f69a715... 2024-01-15T22:25:55.248 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: disabled 2024-01-15T22:25:55.255 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: inactive 2024-01-15T22:25:55.255 INFO:teuthology.orchestra.run.smithi099.stderr:firewalld.service is not enabled 2024-01-15T22:25:55.255 INFO:teuthology.orchestra.run.smithi099.stderr:Not possible to enable service . firewalld.service is not available 2024-01-15T22:25:55.256 INFO:teuthology.orchestra.run.smithi099.stderr:Waiting for mon to start... 2024-01-15T22:25:55.256 INFO:teuthology.orchestra.run.smithi099.stderr:Waiting for mon... 2024-01-15T22:25:55.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:55 smithi099 bash[34259]: bc29fc4a28c81582b48357087aab5c2f5ec04228ea1bdad7645db1724639ad3d 2024-01-15T22:25:55.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:55 smithi099 systemd[1]: Started Ceph mon.smithi099 for eb17a8d4-b3f4-11ee-95ad-87774f69a715. 2024-01-15T22:25:55.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:55 smithi099 conmon[34300]: cluster 2024-01-15T22:25:55.259645+0000 mon.smithi099 (mon.0) 0 : [INF] mkfs eb17a8d4-b3f4-11ee-95ad-87774f69a715 2024-01-15T22:25:55.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:55 smithi099 conmon[34300]: cluster 2024-01-15T22:25 2024-01-15T22:25:55.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:55 smithi099 conmon[34300]: :55.226343+0000 mon.smithi099 (mon.0) 1 : cluster [INF] mon.smithi099 is new leader, mons smithi099 in quorum (ranks 0) 2024-01-15T22:25:55.727 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: cluster: 2024-01-15T22:25:55.728 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: id: eb17a8d4-b3f4-11ee-95ad-87774f69a715 2024-01-15T22:25:55.728 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-01-15T22:25:55.728 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: 2024-01-15T22:25:55.728 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: services: 2024-01-15T22:25:55.728 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi099 (age 0.481602s) 2024-01-15T22:25:55.728 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: mgr: no daemons active 2024-01-15T22:25:55.728 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-01-15T22:25:55.728 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: 2024-01-15T22:25:55.728 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: data: 2024-01-15T22:25:55.728 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-01-15T22:25:55.728 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-01-15T22:25:55.728 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-01-15T22:25:55.728 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: pgs: 2024-01-15T22:25:55.728 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: 2024-01-15T22:25:56.308 INFO:teuthology.orchestra.run.smithi099.stderr:mon is available 2024-01-15T22:25:56.308 INFO:teuthology.orchestra.run.smithi099.stderr:Assimilating anything we can from ceph.conf... 2024-01-15T22:25:56.517 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:56 smithi099 conmon[34300]: cluster 2024-01-15T22:25:55.246328+0000 mon.smithi099 (mon. 2024-01-15T22:25:56.517 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:56 smithi099 conmon[34300]: 0) 2 : cluster [INF] mon.smithi099 is new leader, mons smithi099 in quorum (ranks 0) 2024-01-15T22:25:56.517 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:56 smithi099 conmon[34300]: cluster 2024-01-15T22:25:55.246633+0000 mon.smithi099 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi099=[v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0]} removed_ranks: {} 2024-01-15T22:25:56.517 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:56 smithi099 conmon[34300]: cluster 2024-01-15T22:25:55.261792+0000 mon.smithi099 (mon.0) 4 : cluster [DBG] fsmap 2024-01-15T22:25:56.517 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:56 smithi099 conmon[34300]: cluster 2024-01-15T22:25:55.264656+0000 mon.smithi099 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-01-15T22:25:56.518 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:56 smithi099 conmon[34300]: cluster 2024-01-15T22:25:55.265140+0000 mon.smithi099 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-01-15T22:25:56.518 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:56 smithi099 conmon[34300]: audit 2024-01-15T22:25:55.727878+0000 mon.smithi099 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.99:0/4091687777' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-01-15T22:25:56.814 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: 2024-01-15T22:25:56.814 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: [global] 2024-01-15T22:25:56.814 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: fsid = eb17a8d4-b3f4-11ee-95ad-87774f69a715 2024-01-15T22:25:56.814 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.99:3300,v1:172.21.15.99:6789] 2024-01-15T22:25:56.814 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-01-15T22:25:56.814 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-01-15T22:25:56.814 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-01-15T22:25:56.814 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-01-15T22:25:56.814 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: 2024-01-15T22:25:56.814 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: [mgr] 2024-01-15T22:25:56.814 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-01-15T22:25:56.815 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: 2024-01-15T22:25:56.815 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: [osd] 2024-01-15T22:25:56.815 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-01-15T22:25:56.815 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-01-15T22:25:57.299 INFO:teuthology.orchestra.run.smithi099.stderr:Generating new minimal ceph.conf... 2024-01-15T22:25:57.516 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:57 smithi099 conmon[34300]: audit 2024-01-15T22:25:56.811377+0000 mon.smithi099 (mon.0) 8 : audit [INF] from='client.? 172.21.15.99:0/1349257078' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-15T22:25:57.517 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:57 smithi099 conmon[34300]: 2024-01-15T22:25:57.517 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:57 smithi099 conmon[34300]: audit 2024-01-15T22:25:56.814148+0000 mon.smithi099 (mon.0) 9 : audit [INF] from='client.? 172.21.15.99:0/1349257078' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-01-15T22:25:58.324 INFO:teuthology.orchestra.run.smithi099.stderr:Restarting the monitor... 2024-01-15T22:25:58.595 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:58 smithi099 conmon[34300]: audit 2024-01-15T22:25:57.810724+0000 mon.smithi099 (mon.0) 10 : audit [DBG] 2024-01-15T22:25:58.595 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:58 smithi099 conmon[34300]: from='client.? 172.21.15.99:0/583648916' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:25:58.595 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:58 smithi099 systemd[1]: Stopping Ceph mon.smithi099 for eb17a8d4-b3f4-11ee-95ad-87774f69a715... 2024-01-15T22:25:58.595 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:58 smithi099 bash[34812]: Error: no container with name or ID ceph-eb17a8d4-b3f4-11ee-95ad-87774f69a715-mon.smithi099 found: no such container 2024-01-15T22:25:58.596 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:58 smithi099 conmon[34300]: debug 2024-01-15T22:25:58.596+0000 7ff60d266700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.smithi099 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-15T22:25:58.596 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:58 smithi099 conmon[34300]: debug 2024-01-15T22:25:58.596+0000 7ff60d266700 -1 mon.smithi099@0(leader) e1 *** Got Signal Terminated *** 2024-01-15T22:25:59.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 bash[34927]: bc29fc4a28c81582b48357087aab5c2f5ec04228ea1bdad7645db1724639ad3d 2024-01-15T22:25:59.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 systemd[1]: ceph-eb17a8d4-b3f4-11ee-95ad-87774f69a715@mon.smithi099.service: Succeeded. 2024-01-15T22:25:59.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 systemd[1]: Stopped Ceph mon.smithi099 for eb17a8d4-b3f4-11ee-95ad-87774f69a715. 2024-01-15T22:25:59.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 systemd[1]: Starting Ceph mon.smithi099 for eb17a8d4-b3f4-11ee-95ad-87774f69a715... 2024-01-15T22:25:59.869 INFO:teuthology.orchestra.run.smithi099.stderr:Setting public_network to 172.21.0.0/20 in mon config section 2024-01-15T22:26:00.104 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.852+0000 7f2d00b86880 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-15T22:26:00.104 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.852+0000 7f2d00b86880 0 ceph version 16.2.14-456-g445562ab (445562ab4bc3ddfb386936119050695810860bcb) pacific (stable), process ceph-mon, pid 7 2024-01-15T22:26:00.104 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.852+0000 7f2d00b86880 0 pidfile_write: ignore empty --pid-file 2024-01-15T22:26:00.105 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 0 load: jerasure load: lrc load: isa 2024-01-15T22:26:00.105 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: RocksDB version: 6.8.1 2024-01-15T22:26:00.105 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: 2024-01-15T22:26:00.105 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-15T22:26:00.105 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Compile date Jan 12 2024 2024-01-15T22:26:00.105 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: DB SUMMARY 2024-01-15T22:26:00.105 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: 2024-01-15T22:26:00.105 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: CURRENT file: CURRENT 2024-01-15T22:26:00.105 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: 2024-01-15T22:26:00.105 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: IDENTITY file: IDENTITY 2024-01-15T22:26:00.105 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: 2024-01-15T22:26:00.105 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-01-15T22:26:00.106 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: 2024-01-15T22:26:00.106 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi099/store.db dir, Total Num: 1, files: 000004.sst 2024-01-15T22:26:00.106 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: 2024-01-15T22:26:00.106 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi099/store.db: 000006.log size: 86505 ; 2024-01-15T22:26:00.106 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: 2024-01-15T22:26:00.106 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.error_if_exists: 0 2024-01-15T22:26:00.106 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.create_if_missing: 0 2024-01-15T22:26:00.106 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.paranoid_checks: 1 2024-01-15T22:26:00.106 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.env: 0x55f46799f080 2024-01-15T22:26:00.106 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.fs: Posix File System 2024-01-15T22:26:00.106 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.info_log: 0x55f468af3d20 2024-01-15T22:26:00.106 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-15T22:26:00.106 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.statistics: (nil) 2024-01-15T22:26:00.106 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.use_fsync: 0 2024-01-15T22:26:00.106 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.max_log_file_size: 0 2024-01-15T22:26:00.107 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-15T22:26:00.107 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-15T22:26:00.107 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-15T22:26:00.107 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-15T22:26:00.107 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.allow_fallocate: 1 2024-01-15T22:26:00.107 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-15T22:26:00.107 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-15T22:26:00.107 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.use_direct_reads: 0 2024-01-15T22:26:00.107 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-15T22:26:00.107 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.create_missing_column_families: 0 2024-01-15T22:26:00.107 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.db_log_dir: 2024-01-15T22:26:00.107 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi099/store.db 2024-01-15T22:26:00.107 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-15T22:26:00.107 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.max_subcompactions: 1 2024-01-15T22:26:00.107 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.max_background_flushes: -1 2024-01-15T22:26:00.108 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-15T22:26:00.108 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-15T22:26:00.108 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-15T22:26:00.108 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-15T22:26:00.108 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-15T22:26:00.108 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.advise_random_on_open: 1 2024-01-15T22:26:00.108 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-15T22:26:00.108 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.write_buffer_manager: 0x55f46974a480 2024-01-15T22:26:00.108 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-15T22:26:00.108 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-15T22:26:00.108 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-15T22:26:00.108 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-15T22:26:00.108 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.rate_limiter: (nil) 2024-01-15T22:26:00.108 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-15T22:26:00.109 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-15T22:26:00.109 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-15T22:26:00.109 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-15T22:26:00.109 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.unordered_write: 0 2024-01-15T22:26:00.109 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-15T22:26:00.109 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-15T22:26:00.109 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-15T22:26:00.109 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-15T22:26:00.109 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.row_cache: None 2024-01-15T22:26:00.109 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.wal_filter: None 2024-01-15T22:26:00.109 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-15T22:26:00.109 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-15T22:26:00.109 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.preserve_deletes: 0 2024-01-15T22:26:00.109 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.two_write_queues: 0 2024-01-15T22:26:00.109 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.manual_wal_flush: 0 2024-01-15T22:26:00.110 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.atomic_flush: 0 2024-01-15T22:26:00.110 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-15T22:26:00.110 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-15T22:26:00.110 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-15T22:26:00.110 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.log_readahead_size: 0 2024-01-15T22:26:00.110 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-15T22:26:00.110 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.max_background_jobs: 2 2024-01-15T22:26:00.110 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.max_background_compactions: -1 2024-01-15T22:26:00.110 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-15T22:26:00.110 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-15T22:26:00.110 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-15T22:26:00.110 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.max_total_wal_size: 0 2024-01-15T22:26:00.110 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-15T22:26:00.110 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-15T22:26:00.111 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-15T22:26:00.111 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-15T22:26:00.111 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.max_open_files: -1 2024-01-15T22:26:00.111 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.bytes_per_sync: 0 2024-01-15T22:26:00.111 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-15T22:26:00.111 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-15T22:26:00.111 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-15T22:26:00.111 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Compression algorithms supported: 2024-01-15T22:26:00.111 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-15T22:26:00.111 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: kZSTD supported: 0 2024-01-15T22:26:00.111 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: kXpressCompression supported: 0 2024-01-15T22:26:00.111 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-15T22:26:00.111 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: kLZ4Compression supported: 1 2024-01-15T22:26:00.111 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: kBZip2Compression supported: 0 2024-01-15T22:26:00.112 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: kZlibCompression supported: 1 2024-01-15T22:26:00.112 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: kSnappyCompression supported: 1 2024-01-15T22:26:00.112 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.856+0000 7f2d00b86880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-15T22:26:00.112 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi099/store.db/MANIFEST-000005 2024-01-15T22:26:00.112 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: 2024-01-15T22:26:00.112 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-15T22:26:00.112 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: 2024-01-15T22:26:00.112 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-15T22:26:00.112 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.merge_operator: 2024-01-15T22:26:00.112 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.compaction_filter: None 2024-01-15T22:26:00.112 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.compaction_filter_factory: None 2024-01-15T22:26:00.112 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-15T22:26:00.112 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-15T22:26:00.112 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f468a9dd28) 2024-01-15T22:26:00.112 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: cache_index_and_filter_blocks: 1 2024-01-15T22:26:00.113 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-15T22:26:00.113 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-15T22:26:00.113 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: pin_top_level_index_and_filter: 1 2024-01-15T22:26:00.113 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: index_type: 0 2024-01-15T22:26:00.113 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: data_block_index_type: 0 2024-01-15T22:26:00.113 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: index_shortening: 1 2024-01-15T22:26:00.113 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: data_block_hash_table_util_ratio: 0.750000 2024-01-15T22:26:00.113 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: hash_index_allow_collision: 1 2024-01-15T22:26:00.113 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: checksum: 1 2024-01-15T22:26:00.113 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: no_block_cache: 0 2024-01-15T22:26:00.113 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: block_cache: 0x55f468ad6f10 2024-01-15T22:26:00.113 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: block_cache_name: BinnedLRUCache 2024-01-15T22:26:00.113 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: block_cache_options: 2024-01-15T22:26:00.113 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: capacity : 536870912 2024-01-15T22:26:00.113 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: num_shard_bits : 4 2024-01-15T22:26:00.114 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: strict_capacity_limit : 0 2024-01-15T22:26:00.114 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: high_pri_pool_ratio: 0.000 2024-01-15T22:26:00.114 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: block_cache_compressed: (nil) 2024-01-15T22:26:00.114 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: persistent_cache: (nil) 2024-01-15T22:26:00.114 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: block_size: 4096 2024-01-15T22:26:00.114 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: block_size_deviation: 10 2024-01-15T22:26:00.114 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: block_restart_interval: 16 2024-01-15T22:26:00.114 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: index_block_restart_interval: 1 2024-01-15T22:26:00.114 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: metadata_block_size: 4096 2024-01-15T22:26:00.114 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: partition_filters: 0 2024-01-15T22:26:00.114 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: use_delta_encoding: 1 2024-01-15T22:26:00.114 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-15T22:26:00.114 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: whole_key_filtering: 1 2024-01-15T22:26:00.114 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: verify_compression: 0 2024-01-15T22:26:00.114 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: read_amp_bytes_per_bit: 0 2024-01-15T22:26:00.115 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: format_version: 2 2024-01-15T22:26:00.115 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: enable_index_compression: 1 2024-01-15T22:26:00.115 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: block_align: 0 2024-01-15T22:26:00.115 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: 2024-01-15T22:26:00.115 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-15T22:26:00.115 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-15T22:26:00.115 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.compression: NoCompression 2024-01-15T22:26:00.115 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-15T22:26:00.115 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-15T22:26:00.115 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-15T22:26:00.115 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.num_levels: 7 2024-01-15T22:26:00.115 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-15T22:26:00.115 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-15T22:26:00.115 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-15T22:26:00.116 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-15T22:26:00.116 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-15T22:26:00.116 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-15T22:26:00.116 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-15T22:26:00.116 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-15T22:26:00.116 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-15T22:26:00.116 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-15T22:26:00.116 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.compression_opts.level: 32767 2024-01-15T22:26:00.116 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-15T22:26:00.116 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-15T22:26:00.116 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-15T22:26:00.116 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.compression_opts.enabled: false 2024-01-15T22:26:00.116 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-15T22:26:00.116 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-15T22:26:00.116 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-15T22:26:00.117 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-15T22:26:00.117 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-15T22:26:00.117 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-15T22:26:00.117 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-15T22:26:00.117 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-15T22:26:00.117 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-15T22:26:00.117 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-15T22:26:00.117 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-15T22:26:00.117 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-15T22:26:00.117 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-15T22:26:00.118 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-15T22:26:00.118 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-15T22:26:00.118 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-15T22:26:00.118 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-15T22:26:00.118 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.arena_block_size: 4194304 2024-01-15T22:26:00.118 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-15T22:26:00.118 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-15T22:26:00.119 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-15T22:26:00.119 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-15T22:26:00.119 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-15T22:26:00.119 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-15T22:26:00.119 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-15T22:26:00.119 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-15T22:26:00.119 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-15T22:26:00.119 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-15T22:26:00.119 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-15T22:26:00.119 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-15T22:26:00.119 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-15T22:26:00.119 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-15T22:26:00.119 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.table_properties_collectors: 2024-01-15T22:26:00.119 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.inplace_update_support: 0 2024-01-15T22:26:00.119 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-15T22:26:00.119 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-15T22:26:00.120 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-15T22:26:00.120 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-15T22:26:00.120 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.bloom_locality: 0 2024-01-15T22:26:00.120 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.max_successive_merges: 0 2024-01-15T22:26:00.120 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-15T22:26:00.120 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-15T22:26:00.120 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.force_consistency_checks: 0 2024-01-15T22:26:00.120 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-15T22:26:00.120 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.ttl: 2592000 2024-01-15T22:26:00.120 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-15T22:26:00.120 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi099/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-15T22:26:00.120 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: 2024-01-15T22:26:00.120 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-01-15T22:26:00.120 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: 2024-01-15T22:26:00.121 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 bash[35252]: 8d59a446f27314a7322e7a61fb8d7f434b971fb59ed6d2f67dc9671662edc7eb 2024-01-15T22:26:00.121 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705357559858654, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-01-15T22:26:00.121 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.857+0000 7f2d00b86880 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-01-15T22:26:00.121 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.858+0000 7f2d00b86880 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-01-15T22:26:00.121 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.858+0000 7f2d00b86880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705357559859470, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 83378, "table_properties": {"data_size": 81645, "index_size": 338, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 581, "raw_key_size": 9331, "raw_average_key_size": 47, "raw_value_size": 76157, "raw_average_value_size": 384, "num_data_blocks": 11, "num_entries": 198, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1705357559, "oldest_key_time": 3, "file_creation_time": 0}} 2024-01-15T22:26:00.121 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.858+0000 7f2d00b86880 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-01-15T22:26:00.121 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: 2024-01-15T22:26:00.121 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.859+0000 7f2d00b86880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705357559859882, "job": 1, "event": "recovery_finished"} 2024-01-15T22:26:00.121 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.859+0000 7f2d00b86880 4 rocksdb: DB pointer 0x55f468ba1800 2024-01-15T22:26:00.121 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.859+0000 7f2ce797a700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-15T22:26:00.121 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.859+0000 7f2ce797a700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-15T22:26:00.121 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: ** DB Stats ** 2024-01-15T22:26:00.121 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-15T22:26:00.122 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-15T22:26:00.122 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-15T22:26:00.122 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-15T22:26:00.122 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-15T22:26:00.122 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-15T22:26:00.122 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-15T22:26:00.122 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: 2024-01-15T22:26:00.122 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: ** Compaction Stats [default] ** 2024-01-15T22:26:00.122 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-15T22:26:00.122 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-15T22:26:00.122 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: L0 2/0 83.12 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 147.8 0.00 0.00 1 0.001 0 0 2024-01-15T22:26:00.122 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: Sum 2/0 83.12 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 147.8 0.00 0.00 1 0.001 0 0 2024-01-15T22:26:00.122 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 147.8 0.00 0.00 1 0.001 0 0 2024-01-15T22:26:00.122 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: 2024-01-15T22:26:00.122 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: ** Compaction Stats [default] ** 2024-01-15T22:26:00.122 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-15T22:26:00.123 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-15T22:26:00.123 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 147.8 0.00 0.00 1 0.001 0 0 2024-01-15T22:26:00.123 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-15T22:26:00.123 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-15T22:26:00.123 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-15T22:26:00.123 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: AddFile(Total Files): cumulative 0, interval 0 2024-01-15T22:26:00.123 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-15T22:26:00.123 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: AddFile(Keys): cumulative 0, interval 0 2024-01-15T22:26:00.123 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: Cumulative compaction: 0.00 GB write, 28.81 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-15T22:26:00.123 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: Interval compaction: 0.00 GB write, 28.81 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-15T22:26:00.123 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-15T22:26:00.123 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: 2024-01-15T22:26:00.123 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: ** File Read Latency Histogram By Level [default] ** 2024-01-15T22:26:00.123 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: 2024-01-15T22:26:00.123 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: ** Compaction Stats [default] ** 2024-01-15T22:26:00.124 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-15T22:26:00.124 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-15T22:26:00.124 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: L0 2/0 83.12 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 147.8 0.00 0.00 1 0.001 0 0 2024-01-15T22:26:00.124 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: Sum 2/0 83.12 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 147.8 0.00 0.00 1 0.001 0 0 2024-01-15T22:26:00.124 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-15T22:26:00.124 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: 2024-01-15T22:26:00.124 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: ** Compaction Stats [default] ** 2024-01-15T22:26:00.124 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-15T22:26:00.124 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-15T22:26:00.124 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 147.8 0.00 0.00 1 0.001 0 0 2024-01-15T22:26:00.124 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-15T22:26:00.124 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-15T22:26:00.124 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-15T22:26:00.125 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: AddFile(Total Files): cumulative 0, interval 0 2024-01-15T22:26:00.125 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-15T22:26:00.125 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: AddFile(Keys): cumulative 0, interval 0 2024-01-15T22:26:00.125 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: Cumulative compaction: 0.00 GB write, 28.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-15T22:26:00.125 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-15T22:26:00.125 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-15T22:26:00.125 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: 2024-01-15T22:26:00.125 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: ** File Read Latency Histogram By Level [default] ** 2024-01-15T22:26:00.125 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: 2024-01-15T22:26:00.125 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.860+0000 7f2d00b86880 0 starting mon.smithi099 rank 0 at public addrs [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] at bind addrs [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi099 fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 2024-01-15T22:26:00.125 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.861+0000 7f2d00b86880 1 mon.smithi099@-1(???) e1 preinit fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 2024-01-15T22:26:00.125 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.861+0000 7f2d00b86880 0 mon.smithi099@-1(???).mds e1 new map 2024-01-15T22:26:00.125 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.861+0000 7f2d00b86880 0 mon.smithi099@-1(???).mds e1 print_map 2024-01-15T22:26:00.125 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: e1 2024-01-15T22:26:00.126 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: enable_multiple, ever_enabled_multiple: 1,1 2024-01-15T22:26:00.126 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-15T22:26:00.126 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: legacy client fscid: -1 2024-01-15T22:26:00.126 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: 2024-01-15T22:26:00.126 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: No filesystems configured 2024-01-15T22:26:00.126 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: 2024-01-15T22:26:00.126 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.861+0000 7f2d00b86880 0 mon.smithi099@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-01-15T22:26:00.126 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.861+0000 7f2d00b86880 0 mon.smithi099@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-15T22:26:00.126 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.861+0000 7f2d00b86880 0 mon.smithi099@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-15T22:26:00.126 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.861+0000 7f2d00b86880 0 mon.smithi099@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-15T22:26:00.126 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: audit 2024-01 2024-01-15T22:26:00.126 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: -15T22:25:57.810724+0000 mon.smithi099 (mon.0) 10 : audit [DBG] 2024-01-15T22:26:00.126 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: from='client.? 172.21.15.99:0/583648916' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:26:00.126 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: debug 2024-01-15T22:25:59.861+0000 7f2d00b86880 1 mon.smithi099@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-01-15T22:26:00.126 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 systemd[1]: Started Ceph mon.smithi099 for eb17a8d4-b3f4-11ee-95ad-87774f69a715. 2024-01-15T22:26:00.127 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: cluster 2024-01-15T22:25:59.877547+0000 mon.smithi099 (mon.0) 1 : cluster [INF] mon.smithi099 is new leader, mons smithi099 in quorum (ranks 0) 2024-01-15T22:26:00.127 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: cluster 2024-01-15T22:25:59.877633+0000 mon.smithi099 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi099=[v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0]} removed_ranks: {} 2024-01-15T22:26:00.127 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: cluster 2024-01-15T22:25:59.878283+0000 mon.smithi099 (mon.0) 3 : cluster [DBG] fsmap 2024-01-15T22:26:00.127 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: cluster 2024-01-15T22:25:59.878327+0000 mon.smithi099 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-01-15T22:26:00.127 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:25:59 smithi099 conmon[35292]: cluster 2024-01-15T22:25:59.878862+0000 mon.smithi099 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-01-15T22:26:00.815 INFO:teuthology.orchestra.run.smithi099.stderr:Wrote config to /etc/ceph/ceph.conf 2024-01-15T22:26:00.816 INFO:teuthology.orchestra.run.smithi099.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-01-15T22:26:00.816 INFO:teuthology.orchestra.run.smithi099.stderr:Creating mgr... 2024-01-15T22:26:00.816 INFO:teuthology.orchestra.run.smithi099.stderr:Verifying port 9283 ... 2024-01-15T22:26:00.945 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: Failed to reset failed state of unit ceph-eb17a8d4-b3f4-11ee-95ad-87774f69a715@mgr.smithi099.cmjeyn.service: Unit ceph-eb17a8d4-b3f4-11ee-95ad-87774f69a715@mgr.smithi099.cmjeyn.service not loaded. 2024-01-15T22:26:00.951 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: Created symlink /etc/systemd/system/ceph-eb17a8d4-b3f4-11ee-95ad-87774f69a715.target.wants/ceph-eb17a8d4-b3f4-11ee-95ad-87774f69a715@mgr.smithi099.cmjeyn.service → /etc/systemd/system/ceph-eb17a8d4-b3f4-11ee-95ad-87774f69a715@.service. 2024-01-15T22:26:01.603 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: disabled 2024-01-15T22:26:01.610 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: inactive 2024-01-15T22:26:01.610 INFO:teuthology.orchestra.run.smithi099.stderr:firewalld.service is not enabled 2024-01-15T22:26:01.610 INFO:teuthology.orchestra.run.smithi099.stderr:Not possible to enable service . firewalld.service is not available 2024-01-15T22:26:01.618 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: disabled 2024-01-15T22:26:01.625 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: inactive 2024-01-15T22:26:01.626 INFO:teuthology.orchestra.run.smithi099.stderr:firewalld.service is not enabled 2024-01-15T22:26:01.626 INFO:teuthology.orchestra.run.smithi099.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-01-15T22:26:01.627 INFO:teuthology.orchestra.run.smithi099.stderr:Waiting for mgr to start... 2024-01-15T22:26:01.627 INFO:teuthology.orchestra.run.smithi099.stderr:Waiting for mgr... 2024-01-15T22:26:01.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:01 smithi099 conmon[35292]: audit 2024-01-15T22:26:00.377841+0000 mon.smithi099 (mon.0) 6 : audit [INF] from='client.? 172.21.15.99:0/4188022252' entity='client.admin' 2024-01-15T22:26:01.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:01 smithi099 conmon[35292]: 2024-01-15T22:26:02.179 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: 2024-01-15T22:26:02.179 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: { 2024-01-15T22:26:02.179 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "fsid": "eb17a8d4-b3f4-11ee-95ad-87774f69a715", 2024-01-15T22:26:02.179 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "health": { 2024-01-15T22:26:02.179 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-15T22:26:02.179 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "checks": {}, 2024-01-15T22:26:02.180 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "mutes": [] 2024-01-15T22:26:02.180 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-15T22:26:02.180 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-15T22:26:02.180 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "quorum": [ 2024-01-15T22:26:02.180 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: 0 2024-01-15T22:26:02.180 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: ], 2024-01-15T22:26:02.180 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-15T22:26:02.180 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "smithi099" 2024-01-15T22:26:02.180 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: ], 2024-01-15T22:26:02.180 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-01-15T22:26:02.180 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "monmap": { 2024-01-15T22:26:02.181 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-15T22:26:02.181 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-15T22:26:02.181 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-15T22:26:02.181 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-15T22:26:02.181 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "osdmap": { 2024-01-15T22:26:02.181 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-15T22:26:02.181 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-15T22:26:02.181 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-15T22:26:02.181 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-15T22:26:02.185 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-15T22:26:02.185 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-15T22:26:02.185 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-15T22:26:02.186 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-15T22:26:02.186 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "pgmap": { 2024-01-15T22:26:02.186 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-15T22:26:02.186 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-15T22:26:02.186 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-15T22:26:02.186 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-15T22:26:02.186 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-15T22:26:02.186 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-15T22:26:02.186 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-15T22:26:02.186 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-15T22:26:02.186 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-15T22:26:02.186 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "fsmap": { 2024-01-15T22:26:02.186 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-15T22:26:02.187 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-15T22:26:02.187 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-15T22:26:02.187 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-15T22:26:02.187 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-15T22:26:02.187 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "available": false, 2024-01-15T22:26:02.187 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-15T22:26:02.187 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "modules": [ 2024-01-15T22:26:02.187 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "iostat", 2024-01-15T22:26:02.187 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "nfs", 2024-01-15T22:26:02.187 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "restful" 2024-01-15T22:26:02.187 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: ], 2024-01-15T22:26:02.187 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "services": {} 2024-01-15T22:26:02.188 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-15T22:26:02.188 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "servicemap": { 2024-01-15T22:26:02.188 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-15T22:26:02.188 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "modified": "2024-01-15T22:25:55.260083+0000", 2024-01-15T22:26:02.188 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "services": {} 2024-01-15T22:26:02.188 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-15T22:26:02.188 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-15T22:26:02.188 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: } 2024-01-15T22:26:02.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:02 smithi099 conmon[35292]: audit 2024-01-15T22:26:02.176995+0000 mon.smithi099 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.99:0/861340816' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-15T22:26:02.707 INFO:teuthology.orchestra.run.smithi099.stderr:mgr not available, waiting (1/15)... 2024-01-15T22:26:05.218 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: 2024-01-15T22:26:05.218 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: { 2024-01-15T22:26:05.218 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "fsid": "eb17a8d4-b3f4-11ee-95ad-87774f69a715", 2024-01-15T22:26:05.219 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "health": { 2024-01-15T22:26:05.219 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-15T22:26:05.219 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "checks": {}, 2024-01-15T22:26:05.219 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "mutes": [] 2024-01-15T22:26:05.219 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-15T22:26:05.219 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-15T22:26:05.219 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "quorum": [ 2024-01-15T22:26:05.219 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: 0 2024-01-15T22:26:05.219 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: ], 2024-01-15T22:26:05.219 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-15T22:26:05.219 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "smithi099" 2024-01-15T22:26:05.219 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: ], 2024-01-15T22:26:05.219 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "quorum_age": 5, 2024-01-15T22:26:05.219 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "monmap": { 2024-01-15T22:26:05.219 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-15T22:26:05.220 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-15T22:26:05.220 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-15T22:26:05.220 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-15T22:26:05.220 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "osdmap": { 2024-01-15T22:26:05.220 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-15T22:26:05.220 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-15T22:26:05.220 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-15T22:26:05.220 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-15T22:26:05.223 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-15T22:26:05.223 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-15T22:26:05.224 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-15T22:26:05.224 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-15T22:26:05.224 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "pgmap": { 2024-01-15T22:26:05.224 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-15T22:26:05.224 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-15T22:26:05.224 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-15T22:26:05.224 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-15T22:26:05.224 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-15T22:26:05.224 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-15T22:26:05.224 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-15T22:26:05.224 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-15T22:26:05.224 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-15T22:26:05.224 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "fsmap": { 2024-01-15T22:26:05.225 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-15T22:26:05.225 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-15T22:26:05.225 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-15T22:26:05.225 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-15T22:26:05.225 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-15T22:26:05.225 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "available": false, 2024-01-15T22:26:05.225 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-15T22:26:05.225 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "modules": [ 2024-01-15T22:26:05.225 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "iostat", 2024-01-15T22:26:05.225 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "nfs", 2024-01-15T22:26:05.225 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "restful" 2024-01-15T22:26:05.225 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: ], 2024-01-15T22:26:05.225 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "services": {} 2024-01-15T22:26:05.225 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-15T22:26:05.225 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "servicemap": { 2024-01-15T22:26:05.226 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-15T22:26:05.226 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "modified": "2024-01-15T22:25:55.260083+0000", 2024-01-15T22:26:05.226 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "services": {} 2024-01-15T22:26:05.226 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-15T22:26:05.226 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-15T22:26:05.226 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: } 2024-01-15T22:26:05.673 INFO:teuthology.orchestra.run.smithi099.stderr:mgr not available, waiting (2/15)... 2024-01-15T22:26:05.946 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:05 smithi099 conmon[35292]: audit 2024-01-15T22:26:05.217296+0000 mon.smithi099 (mon.0) 2024-01-15T22:26:05.946 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:05 smithi099 conmon[35292]: 8 : audit [DBG] from='client.? 172.21.15.99:0/3786612257' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-15T22:26:07.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:07 smithi099 conmon[35292]: cluster 2024-01-15T22:26:06.989616+0000 mon.smithi099 (mon.0) 9 : cluster [INF] 2024-01-15T22:26:07.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:07 smithi099 conmon[35292]: Activating manager daemon smithi099.cmjeyn 2024-01-15T22:26:07.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:07 smithi099 conmon[35292]: cluster 2024-01-15T22:26:06.992071+0000 mon.smithi099 ( 2024-01-15T22:26:07.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:07 smithi099 conmon[35292]: mon.0) 10 : cluster [DBG] mgrmap e2: smithi099.cmjeyn(active, starting, since 0.00253801s) 2024-01-15T22:26:07.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:07 smithi099 conmon[35292]: audit 2024-01-15T22:26: 2024-01-15T22:26:07.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:07 smithi099 conmon[35292]: 06.992772+0000 mon.smithi099 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.99:0/1172912160' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-15T22:26:07.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:07 smithi099 conmon[35292]: audit 2024-01-15T22:26:07.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:07 smithi099 conmon[35292]: 2024-01-15T22:26:06.992970+0000 mon.smithi099 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.99:0/1172912160' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-15T22:26:07.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:07 smithi099 conmon[35292]: audit 2024-01-15T22:26: 2024-01-15T22:26:07.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:07 smithi099 conmon[35292]: 06.993144+0000 mon.smithi099 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.99:0/1172912160' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-15T22:26:07.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:07 smithi099 conmon[35292]: audit 2024-01-15T22:26:06.993311 2024-01-15T22:26:07.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:07 smithi099 conmon[35292]: +0000 mon.smithi099 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.99:0/1172912160' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-01-15T22:26:07.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:07 smithi099 conmon[35292]: audit 2024-01-15T22:26:06.993500+0000 mon.smithi099 (mon 2024-01-15T22:26:07.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:07 smithi099 conmon[35292]: .0) 15 : audit [DBG] from='mgr.14100 172.21.15.99:0/1172912160' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mgr metadata", "who": "smithi099.cmjeyn", "id": "smithi099.cmjeyn"}]: dispatch 2024-01-15T22:26:07.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:07 smithi099 conmon[35292]: cluster 2024- 2024-01-15T22:26:07.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:07 smithi099 conmon[35292]: 01-15T22:26:07.007494+0000 mon.smithi099 (mon.0) 16 : cluster [INF] Manager daemon smithi099.cmjeyn is now available 2024-01-15T22:26:07.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:07 smithi099 conmon[35292]: audit 2024-01-15T22:26:07.018156+0000 2024-01-15T22:26:07.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:07 smithi099 conmon[35292]: mon.smithi099 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.99:0/1172912160' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:26:07.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:07 smithi099 conmon[35292]: audit 2024-01-15T22:26:07.021748+0000 mon.smithi099 (mon. 2024-01-15T22:26:07.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:07 smithi099 conmon[35292]: 0) 18 : audit [INF] from='mgr.14100 172.21.15.99:0/1172912160' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi099.cmjeyn/mirror_snapshot_schedule"}]: dispatch 2024-01-15T22:26:07.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:07 smithi099 conmon[35292]: audit 2024 2024-01-15T22:26:07.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:07 smithi099 conmon[35292]: -01-15T22:26:07.024457+0000 mon.smithi099 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.99:0/1172912160' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi099.cmjeyn/trash_purge_schedule"}]: dispatch 2024-01-15T22:26:07.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:07 smithi099 conmon[35292]: audit 2024-01-15T22:26:07.029169 2024-01-15T22:26:07.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:07 smithi099 conmon[35292]: +0000 mon.smithi099 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.99:0/1172912160' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:26:07.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:07 smithi099 conmon[35292]: audit 2024-01-15T22:26:07.034003+0000 mon.smithi099 (mon 2024-01-15T22:26:07.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:07 smithi099 conmon[35292]: .0) 21 : audit [INF] from='mgr.14100 172.21.15.99:0/1172912160' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:26:08.348 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: 2024-01-15T22:26:08.348 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: { 2024-01-15T22:26:08.348 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "fsid": "eb17a8d4-b3f4-11ee-95ad-87774f69a715", 2024-01-15T22:26:08.348 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "health": { 2024-01-15T22:26:08.348 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-15T22:26:08.348 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "checks": {}, 2024-01-15T22:26:08.348 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "mutes": [] 2024-01-15T22:26:08.348 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-15T22:26:08.348 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-15T22:26:08.348 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "quorum": [ 2024-01-15T22:26:08.349 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: 0 2024-01-15T22:26:08.349 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: ], 2024-01-15T22:26:08.349 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-15T22:26:08.349 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "smithi099" 2024-01-15T22:26:08.349 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: ], 2024-01-15T22:26:08.349 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "quorum_age": 8, 2024-01-15T22:26:08.349 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "monmap": { 2024-01-15T22:26:08.349 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-15T22:26:08.349 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-15T22:26:08.349 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-15T22:26:08.349 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-15T22:26:08.349 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "osdmap": { 2024-01-15T22:26:08.349 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-15T22:26:08.349 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-15T22:26:08.349 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-15T22:26:08.350 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-15T22:26:08.353 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-15T22:26:08.353 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-15T22:26:08.353 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-15T22:26:08.353 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-15T22:26:08.353 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "pgmap": { 2024-01-15T22:26:08.353 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-15T22:26:08.353 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-15T22:26:08.353 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-15T22:26:08.353 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-15T22:26:08.353 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-15T22:26:08.353 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-15T22:26:08.353 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-15T22:26:08.354 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-15T22:26:08.354 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-15T22:26:08.354 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "fsmap": { 2024-01-15T22:26:08.354 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-15T22:26:08.354 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-15T22:26:08.354 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-15T22:26:08.354 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-15T22:26:08.354 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-15T22:26:08.354 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "available": true, 2024-01-15T22:26:08.354 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-15T22:26:08.354 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "modules": [ 2024-01-15T22:26:08.354 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "iostat", 2024-01-15T22:26:08.354 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "nfs", 2024-01-15T22:26:08.354 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "restful" 2024-01-15T22:26:08.354 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: ], 2024-01-15T22:26:08.354 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "services": {} 2024-01-15T22:26:08.354 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-15T22:26:08.355 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "servicemap": { 2024-01-15T22:26:08.355 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-15T22:26:08.355 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "modified": "2024-01-15T22:25:55.260083+0000", 2024-01-15T22:26:08.355 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "services": {} 2024-01-15T22:26:08.355 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-15T22:26:08.355 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-15T22:26:08.355 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: } 2024-01-15T22:26:08.830 INFO:teuthology.orchestra.run.smithi099.stderr:mgr is available 2024-01-15T22:26:09.097 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:08 smithi099 conmon[35292]: cluster 2024-01-15T22:26:07.994626+0000 mon.smithi099 (mon. 2024-01-15T22:26:09.097 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:08 smithi099 conmon[35292]: 0) 22 : cluster [DBG] mgrmap e3: smithi099.cmjeyn(active, since 1.00509s) 2024-01-15T22:26:09.097 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:08 smithi099 conmon[35292]: audit 2024-01-15T22:26:08.346792+0000 mon.smithi099 (mon.0) 23 : audit [DBG] from='client.? 172.21.15.99:0/1305981941' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-15T22:26:09.433 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: 2024-01-15T22:26:09.433 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: [global] 2024-01-15T22:26:09.433 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: fsid = eb17a8d4-b3f4-11ee-95ad-87774f69a715 2024-01-15T22:26:09.433 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-01-15T22:26:09.433 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-01-15T22:26:09.433 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-01-15T22:26:09.433 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-01-15T22:26:09.434 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: 2024-01-15T22:26:09.434 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: [mgr] 2024-01-15T22:26:09.434 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-01-15T22:26:09.434 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: 2024-01-15T22:26:09.434 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: [osd] 2024-01-15T22:26:09.434 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-01-15T22:26:09.434 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-01-15T22:26:09.930 INFO:teuthology.orchestra.run.smithi099.stderr:Enabling cephadm module... 2024-01-15T22:26:10.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:09 smithi099 conmon[35292]: audit 2024-01-15T22:26:09.432563+0000 mon.smithi099 (mon.0) 24 2024-01-15T22:26:10.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:09 smithi099 conmon[35292]: : audit [INF] from='client.? 172.21.15.99:0/1997120240' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-15T22:26:11.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:10 smithi099 conmon[35292]: cluster 2024-01-15T22:26:09.994748+0000 mon.smithi099 (mon.0) 25 : cluster [DBG] mgrmap e4: smithi099.cmjeyn(active, since 3s) 2024-01-15T22:26:11.407 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:10 smithi099 conmon[35292]: audit 2024-01-15T22:26:10.516260+0000 mon.smithi099 (mon.0) 26 : audit [INF] from='client.? 172.21.15.99:0/1599368131' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-01-15T22:26:12.226 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: { 2024-01-15T22:26:12.226 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "epoch": 5, 2024-01-15T22:26:12.226 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "available": true, 2024-01-15T22:26:12.226 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "active_name": "smithi099.cmjeyn", 2024-01-15T22:26:12.226 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_standby": 0 2024-01-15T22:26:12.226 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: } 2024-01-15T22:26:12.249 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:11 smithi099 conmon[35292]: audit 2024-01-15T22:26:10.999767+0000 mon.smithi099 (mon.0) 27 : audit [INF] 2024-01-15T22:26:12.249 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:11 smithi099 conmon[35292]: from='client.? 172.21.15.99:0/1599368131' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-01-15T22:26:12.249 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:11 smithi099 conmon[35292]: cluster 2024-01-15T22:26:10.999831+0000 mon.smithi099 (mon.0) 28 : cluster [DBG] mgrmap e5: smithi099.cmjeyn(active, since 4s) 2024-01-15T22:26:12.696 INFO:teuthology.orchestra.run.smithi099.stderr:Waiting for the mgr to restart... 2024-01-15T22:26:12.696 INFO:teuthology.orchestra.run.smithi099.stderr:Waiting for mgr epoch 5... 2024-01-15T22:26:13.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:12 smithi099 conmon[35292]: audit 2024-01-15T22:26:12.226822+0000 mon.smithi099 (mon. 2024-01-15T22:26:13.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:12 smithi099 conmon[35292]: 0) 29 : audit [DBG] from='client.? 172.21.15.99:0/2329712798' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-15T22:26:16.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:16 smithi099 conmon[35292]: cluster 2024-01-15T22:26:16.374670+0000 mon.smithi099 (mon.0) 30 : cluster 2024-01-15T22:26:16.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:16 smithi099 conmon[35292]: [INF] Active manager daemon smithi099.cmjeyn restarted 2024-01-15T22:26:16.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:16 smithi099 conmon[35292]: cluster 2024-01-15T22:26:16.374971+0000 mon.smithi099 (mon.0) 31 : cluster [INF] Activating manager daemon smithi099.cmjeyn 2024-01-15T22:26:16.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:16 smithi099 conmon[35292]: cluster 2024-01-15T22:26:16.377689 2024-01-15T22:26:16.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:16 smithi099 conmon[35292]: +0000 mon.smithi099 (mon.0) 32 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-01-15T22:26:16.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:16 smithi099 conmon[35292]: cluster 2024-01-15 2024-01-15T22:26:16.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:16 smithi099 conmon[35292]: T22:26:16.377791+0000 mon.smithi099 2024-01-15T22:26:16.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:16 smithi099 conmon[35292]: (mon.0) 33 : cluster [DBG] 2024-01-15T22:26:16.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:16 smithi099 conmon[35292]: mgrmap e6: smithi099.cmjeyn(active, starting, since 0.0029329s) 2024-01-15T22:26:16.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:16 smithi099 conmon[35292]: audit 2024-01 2024-01-15T22:26:16.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:16 smithi099 conmon[35292]: -15T22:26:16.378855+0000 2024-01-15T22:26:16.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:16 smithi099 conmon[35292]: mon.smithi099 (mon.0) 34 : audit 2024-01-15T22:26:16.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:16 smithi099 conmon[35292]: [DBG] from='mgr.14116 172.21.15.99:0/2568445624' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-01-15T22:26:16.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:16 smithi099 conmon[35292]: audit 2024-01-15T22:26:16.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:16 smithi099 conmon[35292]: 2024-01-15T22:26: 2024-01-15T22:26:16.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:16 smithi099 conmon[35292]: 16.379394+0000 mon.smithi099 (mon. 2024-01-15T22:26:16.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:16 smithi099 conmon[35292]: 0) 35 : audit [DBG] from='mgr.14116 172.21.15.99:0/2568445624' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mgr metadata", "who": "smithi099.cmjeyn", "id": "smithi099.cmjeyn"}]: dispatch 2024-01-15T22:26:16.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:16 smithi099 conmon[35292]: audit 2024-01 2024-01-15T22:26:16.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:16 smithi099 conmon[35292]: -15T22:26:16.380189 2024-01-15T22:26:16.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:16 smithi099 conmon[35292]: +0000 mon.smithi099 (mon.0) 36 : 2024-01-15T22:26:16.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:16 smithi099 conmon[35292]: audit [DBG] from='mgr.14116 172.21.15.99:0/2568445624' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-15T22:26:16.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:16 smithi099 conmon[35292]: audit 2024 2024-01-15T22:26:16.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:16 smithi099 conmon[35292]: -01-15T22:26: 2024-01-15T22:26:16.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:16 smithi099 conmon[35292]: 16.380417+0000 mon.smithi099 (mon.0 2024-01-15T22:26:16.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:16 smithi099 conmon[35292]: ) 37 : audit [DBG] from='mgr.14116 172.21.15.99:0/2568445624' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-15T22:26:16.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:16 smithi099 conmon[35292]: audit 2024-01 2024-01-15T22:26:16.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:16 smithi099 conmon[35292]: -15T22:26:16.380632+0000 2024-01-15T22:26:16.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:16 smithi099 conmon[35292]: mon.smithi099 (mon.0) 38 : audit 2024-01-15T22:26:16.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:16 smithi099 conmon[35292]: [DBG] from='mgr.14116 172.21.15.99:0/2568445624' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-15T22:26:16.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:16 smithi099 conmon[35292]: cluster 2024-01-15T22:26:16.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:16 smithi099 conmon[35292]: 2024-01-15T22:26:16 2024-01-15T22:26:16.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:16 smithi099 conmon[35292]: .382479+0000 mon.smithi099 (mon.0) 2024-01-15T22:26:16.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:16 smithi099 conmon[35292]: 39 : cluster [INF] Manager daemon smithi099.cmjeyn is now available 2024-01-15T22:26:16.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:16 smithi099 conmon[35292]: audit 2024- 2024-01-15T22:26:16.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:16 smithi099 conmon[35292]: 01-15T22:26:16.394321 2024-01-15T22:26:16.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:16 smithi099 conmon[35292]: +0000 mon.smithi099 (mon.0) 40 2024-01-15T22:26:16.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:16 smithi099 conmon[35292]: : audit [INF] from='mgr.14116 172.21.15.99:0/2568445624' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:26:17.386 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: { 2024-01-15T22:26:17.386 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-01-15T22:26:17.386 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "initialized": true 2024-01-15T22:26:17.386 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: } 2024-01-15T22:26:17.879 INFO:teuthology.orchestra.run.smithi099.stderr:mgr epoch 5 is available 2024-01-15T22:26:17.879 INFO:teuthology.orchestra.run.smithi099.stderr:Setting orchestrator backend to cephadm... 2024-01-15T22:26:17.885 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:17 smithi099 conmon[35292]: cephadm 2024-01-15T22:26:16.391755+0000 mgr.smithi099.cmjeyn (mgr.14116) 2024-01-15T22:26:17.885 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:17 smithi099 conmon[35292]: 1 : cephadm [INF] Found migration_current of "None". Setting to last migration. 2024-01-15T22:26:17.885 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:17 smithi099 conmon[35292]: audit 2024-01-15T22:26:16.479757+0000 mon.smithi099 (mon.0) 41 : audit [INF] from='mgr.14116 172.21.15.99:0/2568445624' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:26:17.885 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:17 smithi099 conmon[35292]: audit 2024-01-15T 2024-01-15T22:26:17.885 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:17 smithi099 conmon[35292]: 22:26:16.480770+0000 mon.smithi099 (mon.0) 42 : audit [DBG] from='mgr.14116 172.21.15.99:0/2568445624' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-15T22:26:17.886 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:17 smithi099 conmon[35292]: audit 2024-01-15 2024-01-15T22:26:17.886 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:17 smithi099 conmon[35292]: T22:26:16.482482+0000 mon.smithi099 (mon.0) 43 : audit [DBG] 2024-01-15T22:26:17.886 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:17 smithi099 conmon[35292]: from='mgr.14116 172.21.15.99:0/2568445624' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-15T22:26:17.886 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:17 smithi099 conmon[35292]: audit 2024 2024-01-15T22:26:17.886 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:17 smithi099 conmon[35292]: -01-15T22:26:16.484527+0000 mon.smithi099 (mon.0) 2024-01-15T22:26:17.886 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:17 smithi099 conmon[35292]: 44 : audit [DBG] from='mgr.14116 172.21.15.99:0/2568445624' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-15T22:26:17.886 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:17 smithi099 conmon[35292]: audit 2024-01-15T22:26:16. 2024-01-15T22:26:17.886 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:17 smithi099 conmon[35292]: 496379+0000 mon.smithi099 (mon.0) 45 : audit [INF] 2024-01-15T22:26:17.886 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:17 smithi099 conmon[35292]: from='mgr.14116 172.21.15.99:0/2568445624' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi099.cmjeyn/mirror_snapshot_schedule"}]: dispatch 2024-01-15T22:26:17.886 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:17 smithi099 conmon[35292]: audit 2024- 2024-01-15T22:26:17.886 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:17 smithi099 conmon[35292]: 01-15T22:26:16.499635+0000 mon.smithi099 (mon.0) 2024-01-15T22:26:17.886 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:17 smithi099 conmon[35292]: 46 : audit [INF] from='mgr.14116 172.21.15.99:0/2568445624' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi099.cmjeyn/trash_purge_schedule"}]: dispatch 2024-01-15T22:26:17.886 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:17 smithi099 conmon[35292]: cluster 2024-01-15T22:26 2024-01-15T22:26:17.886 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:17 smithi099 conmon[35292]: :17.380572+0000 mon.smithi099 (mon.0) 47 : cluster 2024-01-15T22:26:17.887 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:17 smithi099 conmon[35292]: [DBG] mgrmap e7: smithi099.cmjeyn(active, since 1.00571s) 2024-01-15T22:26:18.903 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:18 smithi099 conmon[35292]: audit 2024-01-15T22:26:17.379684 2024-01-15T22:26:18.903 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:18 smithi099 conmon[35292]: +0000 mgr.smithi099.cmjeyn (mgr.14116) 2 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-15T22:26:18.903 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:18 smithi099 conmon[35292]: audit 2024-01-15T22:26:17.386710+0000 mgr.smithi099.cmjeyn (mgr.14116) 3 : audit 2024-01-15T22:26:18.903 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:18 smithi099 conmon[35292]: [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-15T22:26:19.581 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: value unchanged 2024-01-15T22:26:19.760 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:19 smithi099 conmon[35292]: audit 2024-01-15T22:26:18.477442+0000 mgr.smithi099.cmjeyn (mgr.14116) 4 : 2024-01-15T22:26:19.760 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:19 smithi099 conmon[35292]: audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-01-15T22:26:19.760 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:19 smithi099 conmon[35292]: audit 2024-01-15T22:26:18.481382+0000 mon.smithi099 (mon.0) 48 : audit [INF] from='mgr.14116 172.21.15.99:0/2568445624' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:26:19.760 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:19 smithi099 conmon[35292]: cluster 2024-01-15T22:26:18.484938+0000 mon.smithi099 (mon.0) 49 : cluster [DBG] mgrmap e8: smithi099.cmjeyn(active, since 2s) 2024-01-15T22:26:19.760 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:19 smithi099 conmon[35292]: audit 2024-01-15T22:26:18.508341+0000 mon.smithi099 (mon.0) 50 : audit [DBG] from='mgr.14116 172.21.15.99:0/2568445624' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-15T22:26:20.053 INFO:teuthology.orchestra.run.smithi099.stderr:Generating ssh key... 2024-01-15T22:26:21.153 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:20 smithi099 conmon[35292]: audit 2024-01-15T22:26:19.581986+0000 mgr.smithi099.cmjeyn (mgr.14116) 2024-01-15T22:26:21.153 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:20 smithi099 conmon[35292]: 5 : audit [DBG] from='client.14128 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-01-15T22:26:22.096 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCjTI9efurZE+QaaqGWYJLXbARGE72hpNNsu5b3cjdlHQdaDrW8EVlM2erTrkiSyh81ULv9XKu5LmuifrZnU5uCcrFr/GFTq5q0BXP+OToB89dz6xKnmdm1Vdi+LZlV7Xdi6+2mFA5n3xT7r6zIpyCTp+Yy33pjbcZT0379Zr//I0vyu3gG0cZtAE/KyhvyHIFTCliWtLw3378w/T72mwnzHaTC8N8hqkKYWeeGpfmlhgkkZizX16iqjJJd956/1Q6kkFxh6Xje5xvmOlY3nFWIabecbZEvjouWXSjIJ2m6Mt+KjBkGpYOBzzpf3GrIM2HXF0djaGh9/f1gXgSC/aQyKo6v8Fne1zN6I+QSnfxN3auRYMDnaygvUFDy4uc4Xu9IK6kVqUmkyWznsAr5PvQYEaxHC/2f8+kmItQK73Ws8B8hTI2x6HOugIGn8EgHc3K1pzswX6Xsu7ftxDLnisENGI+7/W0CUyeppN4jphwFYmDQKoLA3eFVNKXrEpJGVXE= ceph-eb17a8d4-b3f4-11ee-95ad-87774f69a715 2024-01-15T22:26:22.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:22 smithi099 conmon[35292]: audit 2024-01-15T22:26:20.648838+0000 mgr.smithi099.cmjeyn (mgr.14116) 6 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-01-15T22:26:22.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:22 smithi099 conmon[35292]: cephadm 2024-01-15T22:26:20.649082+0000 mgr.smithi099.cmjeyn (mgr.14116) 7 : cephadm [INF] Generating ssh key... 2024-01-15T22:26:22.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:22 smithi099 conmon[35292]: audit 2024-01-15T22:26:21.036307+0000 mon.smithi099 (mon.0) 51 : audit [INF] from='mgr.14116 172.21.15.99:0/2568445624' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:26:22.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:22 smithi099 conmon[35292]: audit 2024-01-15T22:26:21.038202+0000 mon.smithi099 (mon.0) 52 : audit [INF] from='mgr.14116 172.21.15.99:0/2568445624' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:26:22.619 INFO:teuthology.orchestra.run.smithi099.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-01-15T22:26:22.620 INFO:teuthology.orchestra.run.smithi099.stderr:Adding key to root@localhost authorized_keys... 2024-01-15T22:26:22.621 INFO:teuthology.orchestra.run.smithi099.stderr:Adding host smithi099... 2024-01-15T22:26:23.153 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:23 smithi099 conmon[35292]: audit 2024-01-15T22:26:22.097129+0000 mgr.smithi099.cmjeyn (mgr.14116) 8 : audit 2024-01-15T22:26:23.153 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:23 smithi099 conmon[35292]: [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-01-15T22:26:23.888 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: Added host 'smithi099' with addr '172.21.15.99' 2024-01-15T22:26:24.410 INFO:teuthology.orchestra.run.smithi099.stderr:Deploying mon service with default placement... 2024-01-15T22:26:25.110 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: Scheduled mon update... 2024-01-15T22:26:25.141 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:24 smithi099 conmon[35292]: audit 2024-01-15T22:26:23.262973+0000 mgr.smithi099.cmjeyn (mgr.14116) 9 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi099", "addr": "172.21.15.99", "target": ["mon-mgr", ""]}]: dispatch 2024-01-15T22:26:25.141 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:24 smithi099 conmon[35292]: audit 2024-01-15T22:26:23.887685+0000 mon.smithi099 (mon.0) 53 : audit [INF] from='mgr.14116 172.21.15.99:0/2568445624' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:26:25.141 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:24 smithi099 conmon[35292]: cephadm 2024-01-15T22:26:23.887983+0000 mgr.smithi099.cmjeyn (mgr.14116) 10 : cephadm [INF] Added host smithi099 2024-01-15T22:26:25.141 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:24 smithi099 conmon[35292]: audit 2024-01-15T22:26:23.891527+0000 mon.smithi099 (mon.0) 54 : audit [DBG] from='mgr.14116 172.21.15.99:0/2568445624' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-15T22:26:26.063 INFO:teuthology.orchestra.run.smithi099.stderr:Deploying mgr service with default placement... 2024-01-15T22:26:26.153 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:26 smithi099 conmon[35292]: audit 2024-01-15T22:26:25.104505+0000 mgr.smithi099.cmjeyn (mgr.14116) 2024-01-15T22:26:26.153 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:26 smithi099 conmon[35292]: 11 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-01-15T22:26:26.153 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:26 smithi099 conmon[35292]: cephadm 2024-01-15T22:26:25.105635+0000 mgr.smithi099.cmjeyn (mgr.14116) 12 : cephadm [INF] Saving service mon spec with placement count:5 2024-01-15T22:26:26.153 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:26 smithi099 conmon[35292]: audit 2024-01-15T22:26:25.110703+0000 mon.smithi099 (mon.0) 55 : audit [INF] from='mgr.14116 172.21.15.99:0/2568445624' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:26:26.794 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-01-15T22:26:27.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:27 smithi099 conmon[35292]: audit 2024-01-15T22:26:26.218552+0000 mon.smithi099 (mon.0) 56 : audit [INF] from='mgr.14116 172.21.15.99:0/2568445624' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:26:27.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:27 smithi099 conmon[35292]: audit 2024-01-15T22:26:26.484844+0000 mon.smithi099 (mon.0) 57 : audit [INF] from='mgr.14116 172.21.15.99:0/2568445624' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:26:27.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:27 smithi099 conmon[35292]: audit 2024-01-15T22:26:26.791475+0000 mgr.smithi099.cmjeyn (mgr.14116) 13 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-01-15T22:26:27.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:27 smithi099 conmon[35292]: cephadm 2024-01-15T22:26:26.792349+0000 mgr.smithi099.cmjeyn (mgr.14116) 14 : cephadm [INF] Saving service mgr spec with placement count:2 2024-01-15T22:26:27.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:27 smithi099 conmon[35292]: audit 2024-01-15T22:26:26.794621+0000 mon.smithi099 (mon.0) 58 : audit [INF] from='mgr.14116 172.21.15.99:0/2568445624' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:26:27.809 INFO:teuthology.orchestra.run.smithi099.stderr:Deploying crash service with default placement... 2024-01-15T22:26:28.463 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: Scheduled crash update... 2024-01-15T22:26:29.095 INFO:teuthology.orchestra.run.smithi099.stderr:Deploying prometheus service with default placement... 2024-01-15T22:26:29.098 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:28 smithi099 conmon[35292]: audit 2024-01-15T22:26:27.809652+0000 mon.smithi099 (mon.0) 59 : audit [INF] from='mgr.14116 172.21.15.99:0/2568445624' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:26:29.098 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:28 smithi099 conmon[35292]: audit 2024-01-15T22:26:28.464075+0000 mon.smithi099 2024-01-15T22:26:29.098 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:28 smithi099 conmon[35292]: (mon.0) 60 : audit [INF] from='mgr.14116 172.21.15.99:0/2568445624' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:26:30.059 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-01-15T22:26:30.087 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:29 smithi099 conmon[35292]: audit 2024-01-15T22:26:28.460225+0000 mgr.smithi099.cmjeyn (mgr.14116) 15 : audit [DBG] from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-01-15T22:26:30.087 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:29 smithi099 conmon[35292]: cephadm 2024-01-15T22:26:28.461193+0000 mgr.smithi099.cmjeyn (mgr.14116) 16 : cephadm [INF] Saving service crash spec with placement * 2024-01-15T22:26:30.550 INFO:teuthology.orchestra.run.smithi099.stderr:Deploying grafana service with default placement... 2024-01-15T22:26:31.177 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-01-15T22:26:31.333 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:31 smithi099 conmon[35292]: audit 2024-01-15T22:26:30.055923+0000 mgr.smithi099.cmjeyn (mgr.14116) 17 : audit [DBG] from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-01-15T22:26:31.333 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:31 smithi099 conmon[35292]: cephadm 2024-01-15T22:26:30 2024-01-15T22:26:31.333 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:31 smithi099 conmon[35292]: .056680+0000 mgr.smithi099.cmjeyn (mgr.14116) 18 : cephadm [INF] Saving service prometheus spec with placement count:1 2024-01-15T22:26:31.333 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:31 smithi099 conmon[35292]: audit 2024-01-15T22:26:30.059573+0000 mon.smithi099 (mon.0) 61 : audit [INF] from='mgr.14116 172.21.15.99:0/2568445624' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:26:31.658 INFO:teuthology.orchestra.run.smithi099.stderr:Deploying node-exporter service with default placement... 2024-01-15T22:26:32.311 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-01-15T22:26:32.457 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:32 smithi099 conmon[35292]: audit 2024-01-15T22:26:31.174085+0000 mgr.smithi099.cmjeyn (mgr.14116) 19 : audit [DBG] from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-01-15T22:26:32.457 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:32 smithi099 conmon[35292]: cephadm 2024-01-15T22:26:31.174913+0000 mgr.smithi099.cmjeyn (mgr.14116) 20 : cephadm [INF] Saving service grafana spec with placement count:1 2024-01-15T22:26:32.457 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:32 smithi099 conmon[35292]: audit 2024-01-15T22:26:31.177758+0000 mon.smithi099 (mon.0) 62 : audit [INF] from='mgr.14116 172.21.15.99:0/2568445624' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:26:32.750 INFO:teuthology.orchestra.run.smithi099.stderr:Deploying alertmanager service with default placement... 2024-01-15T22:26:33.376 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-01-15T22:26:33.565 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:33 smithi099 conmon[35292]: audit 2024-01-15T22:26:32.308086+0000 mgr.smithi099.cmjeyn (mgr.14116) 2024-01-15T22:26:33.565 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:33 smithi099 conmon[35292]: 21 : audit [DBG] from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-01-15T22:26:33.565 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:33 smithi099 conmon[35292]: cephadm 2024-01-15T22:26:32.308971+0000 mgr.smithi099.cmjeyn (mgr.14116) 22 : cephadm [INF] Saving service node-exporter spec with placement * 2024-01-15T22:26:33.565 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:33 smithi099 conmon[35292]: audit 2024-01-15T22: 2024-01-15T22:26:33.565 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:33 smithi099 conmon[35292]: 26:32.311921+0000 mon.smithi099 (mon.0) 63 : audit [INF] from='mgr.14116 172.21.15.99:0/2568445624' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:26:34.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:34 smithi099 conmon[35292]: audit 2024-01-15T22:26:33.374057+0000 mgr.smithi099.cmjeyn (mgr.14116) 23 : audit [DBG] from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-01-15T22:26:34.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:34 smithi099 conmon[35292]: cephadm 2024-01-15T22:26:33 2024-01-15T22:26:34.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:34 smithi099 conmon[35292]: .374896+0000 mgr.smithi099.cmjeyn (mgr.14116) 24 : cephadm [INF] Saving service alertmanager spec with placement count:1 2024-01-15T22:26:34.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:34 smithi099 conmon[35292]: audit 2024-01-15T22:26:33.376947+0000 mon.smithi099 (mon.0) 64 : audit [INF] from='mgr.14116 172.21.15.99:0/2568445624' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:26:35.848 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:35 smithi099 conmon[35292]: audit 2024-01-15T22:26:34.516659+0000 mon.smithi099 (mon.0) 65 : audit [INF] from='client.? 172.21.15.99:0/1366709837' entity='client.admin' 2024-01-15T22:26:36.157 INFO:teuthology.orchestra.run.smithi099.stderr:Enabling the dashboard module... 2024-01-15T22:26:36.903 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:36 smithi099 conmon[35292]: audit 2024-01-15T22:26:35.649700+0000 mon.smithi099 (mon.0) 66 2024-01-15T22:26:36.903 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:36 smithi099 conmon[35292]: : audit [INF] from='client.? 172.21.15.99:0/389991926' entity='client.admin' 2024-01-15T22:26:37.903 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:37 smithi099 conmon[35292]: cluster 2024-01-15T22:26:36.381509+0000 mgr.smithi099.cmjeyn (mgr.14116) 25 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:26:37.903 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:37 smithi099 conmon[35292]: audit 2024-01-15T22:26:36.803614+0000 mon.smithi099 (mon.0) 67 : audit [INF] from='client.? 172.21.15.99:0/664265769' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-01-15T22:26:38.872 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: { 2024-01-15T22:26:38.872 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "epoch": 9, 2024-01-15T22:26:38.872 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "available": true, 2024-01-15T22:26:38.873 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "active_name": "smithi099.cmjeyn", 2024-01-15T22:26:38.873 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_standby": 0 2024-01-15T22:26:38.873 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: } 2024-01-15T22:26:38.903 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:38 smithi099 conmon[35292]: audit 2024-01-15T22:26:37.654719+0000 mon.smithi099 (mon.0) 68 : audit [INF] from='client.? 172.21.15.99:0/664265769' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-01-15T22:26:38.903 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:38 smithi099 conmon[35292]: cluster 2024-01-15T22:26:37.654791+0000 mon.smithi099 (mon.0) 69 : cluster [DBG] mgrmap e9: smithi099.cmjeyn(active, since 21s) 2024-01-15T22:26:39.390 INFO:teuthology.orchestra.run.smithi099.stderr:Waiting for the mgr to restart... 2024-01-15T22:26:39.390 INFO:teuthology.orchestra.run.smithi099.stderr:Waiting for mgr epoch 9... 2024-01-15T22:26:39.903 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:39 smithi099 conmon[35292]: audit 2024-01-15T22:26:38.873057+0000 mon.smithi099 2024-01-15T22:26:39.903 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:39 smithi099 conmon[35292]: (mon.0) 70 : audit [DBG] from='client.? 172.21.15.99:0/4127274492' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-15T22:26:43.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:43 smithi099 conmon[35292]: cluster 2024-01-15T22:26:43.057731+0000 mon.smithi099 (mon.0) 71 2024-01-15T22:26:43.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:43 smithi099 conmon[35292]: : cluster [INF] Active manager daemon smithi099.cmjeyn restarted 2024-01-15T22:26:43.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:43 smithi099 conmon[35292]: cluster 2024-01-15T22:26:43.058043+0000 mon.smithi099 (mon.0) 72 : cluster [INF] Activating manager daemon smithi099.cmjeyn 2024-01-15T22:26:43.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:43 smithi099 conmon[35292]: cluster 2024-01-15T22:26:43.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:43 smithi099 conmon[35292]: 2024-01-15T22:26:43.060974+0000 mon.smithi099 (mon.0) 73 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-01-15T22:26:43.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:43 smithi099 conmon[35292]: cluster 2024-01-15T22 2024-01-15T22:26:43.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:43 smithi099 conmon[35292]: :26:43.061078+0000 mon.smithi099 (mon.0) 74 : 2024-01-15T22:26:43.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:43 smithi099 conmon[35292]: cluster [DBG] mgrmap e10: smithi099.cmjeyn(active, starting, since 0.00316448s) 2024-01-15T22:26:43.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:43 smithi099 conmon[35292]: audit 2024-01-15T22:26:43 2024-01-15T22:26:43.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:43 smithi099 conmon[35292]: .061984+0000 mon.smithi099 (mon.0) 75 : audit 2024-01-15T22:26:43.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:43 smithi099 conmon[35292]: [DBG] from='mgr.14156 172.21.15.99:0/3616479895' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-01-15T22:26:43.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:43 smithi099 conmon[35292]: audit 2024-01-15T22:26:43.062585+0000 2024-01-15T22:26:43.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:43 smithi099 conmon[35292]: mon.smithi099 (mon.0) 76 : audit [DBG] from='mgr.14156 172.21.15.99:0/3616479895' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mgr metadata", "who": "smithi099.cmjeyn", "id": "smithi099.cmjeyn"}]: dispatch 2024-01-15T22:26:43.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:43 smithi099 conmon[35292]: audit 2024- 2024-01-15T22:26:43.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:43 smithi099 conmon[35292]: 01-15T22:26:43.063179+0000 mon.smithi099 (mon 2024-01-15T22:26:43.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:43 smithi099 conmon[35292]: .0) 77 : audit [DBG] from='mgr.14156 172.21.15.99:0/3616479895' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-15T22:26:43.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:43 smithi099 conmon[35292]: audit 2024-01-15T 2024-01-15T22:26:43.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:43 smithi099 conmon[35292]: 22:26:43.063366+0000 mon.smithi099 (mon.0 2024-01-15T22:26:43.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:43 smithi099 conmon[35292]: ) 78 : audit [DBG] from='mgr.14156 172.21.15.99:0/3616479895' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-15T22:26:43.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:43 smithi099 conmon[35292]: audit 2024-01-15T22 2024-01-15T22:26:43.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:43 smithi099 conmon[35292]: :26:43.063508+0000 mon.smithi099 (mon.0) 79 2024-01-15T22:26:43.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:43 smithi099 conmon[35292]: : audit [DBG] from='mgr.14156 172.21.15.99:0/3616479895' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-15T22:26:43.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:43 smithi099 conmon[35292]: cluster 2024-01-15T22:26: 2024-01-15T22:26:43.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:43 smithi099 conmon[35292]: 43.077220+0000 mon.smithi099 (mon.0) 80 : cluster 2024-01-15T22:26:43.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:43 smithi099 conmon[35292]: [INF] Manager daemon smithi099.cmjeyn is now available 2024-01-15T22:26:44.067 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: { 2024-01-15T22:26:44.067 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-01-15T22:26:44.067 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "initialized": true 2024-01-15T22:26:44.067 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: } 2024-01-15T22:26:44.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:44 smithi099 conmon[35292]: audit 2024-01-15T22:26:43.172245+0000 mon.smithi099 (mon. 2024-01-15T22:26:44.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:44 smithi099 conmon[35292]: 0) 81 : audit [DBG] from='mgr.14156 172.21.15.99:0/3616479895' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-15T22:26:44.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:44 smithi099 conmon[35292]: audit 2024-01-15T22:26:43.173690+0000 mon.smithi099 (mon.0) 82 : audit [DBG] from='mgr.14156 172.21.15.99:0/3616479895' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-15T22:26:44.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:44 smithi099 conmon[35292]: audit 2024-01-15T22:26:43.198800+0000 mon.smithi099 (mon.0) 83 : audit [INF] from='mgr.14156 172.21.15.99:0/3616479895' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi099.cmjeyn/mirror_snapshot_schedule"}]: dispatch 2024-01-15T22:26:44.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:44 smithi099 conmon[35292]: audit 2024-01-15T22:26:43.201539+0000 mon.smithi099 (mon.0) 84 : audit [INF] from='mgr.14156 172.21.15.99:0/3616479895' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi099.cmjeyn/trash_purge_schedule"}]: dispatch 2024-01-15T22:26:44.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:44 smithi099 conmon[35292]: cluster 2024-01-15T22:26:44.062452+0000 mon.smithi099 (mon.0) 85 : cluster [DBG] mgrmap e11: smithi099.cmjeyn(active, since 1.00453s) 2024-01-15T22:26:44.597 INFO:teuthology.orchestra.run.smithi099.stderr:mgr epoch 9 is available 2024-01-15T22:26:44.597 INFO:teuthology.orchestra.run.smithi099.stderr:Generating a dashboard self-signed certificate... 2024-01-15T22:26:45.277 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: Self-signed certificate created 2024-01-15T22:26:45.903 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:45 smithi099 conmon[35292]: audit 2024-01-15T22:26:44.062084+0000 mgr.smithi099.cmjeyn (mgr.14156) 1 : audit [DBG] from='client.14160 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-15T22:26:45.903 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:45 smithi099 conmon[35292]: audit 2024-01-15T22:26:44.068054+0000 mgr.smithi099.cmjeyn (mgr.14156) 2 : audit [DBG] from='client.14160 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-15T22:26:45.903 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:45 smithi099 conmon[35292]: audit 2024-01-15T22:26:44.581371+0000 mon.smithi099 (mon.0) 86 : audit [INF] from='mgr.14156 172.21.15.99:0/3616479895' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:26:45.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:45 smithi099 conmon[35292]: audit 2024-01-15T22:26:45.271725+0000 mon.smithi099 (mon.0) 87 : audit [INF] from='mgr.14156 172.21.15.99:0/3616479895' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:26:45.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:45 smithi099 conmon[35292]: audit 2024-01-15T22:26:45.277442+0000 mon.smithi099 (mon.0) 88 : audit [INF] from='mgr.14156 172.21.15.99:0/3616479895' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:26:46.289 INFO:teuthology.orchestra.run.smithi099.stderr:Creating initial admin user... 2024-01-15T22:26:46.903 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:46 smithi099 conmon[35292]: audit 2024-01-15T22:26:45.207003+0000 mgr.smithi099.cmjeyn (mgr. 2024-01-15T22:26:46.903 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:46 smithi099 conmon[35292]: 14156) 3 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-01-15T22:26:46.903 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:46 smithi099 conmon[35292]: cluster 2024-01-15T22:26:45.878047+0000 mon.smithi099 (mon.0) 89 : cluster [DBG] mgrmap e12: smithi099.cmjeyn(active, since 2s) 2024-01-15T22:26:47.101 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$s0F7mYFYstakX.hL9i11W.qzg1d6s9hVEVBl3opwtZrEcjYqoescm", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1705357607, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-01-15T22:26:47.755 INFO:teuthology.orchestra.run.smithi099.stderr:Fetching dashboard port number... 2024-01-15T22:26:48.398 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: 8443 2024-01-15T22:26:48.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:48 smithi099 conmon[35292]: audit 2024-01-15T22:26:46.885277+0000 mgr.smithi099.cmjeyn (mgr.14156) 4 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-01-15T22:26:48.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:48 smithi099 conmon[35292]: audit 2024-01-15T22:26:47.102032+0000 mon.smithi099 (mon.0) 90 : audit [INF] from='mgr.14156 172.21.15.99:0/3616479895' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:26:48.931 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: disabled 2024-01-15T22:26:48.938 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: inactive 2024-01-15T22:26:48.938 INFO:teuthology.orchestra.run.smithi099.stderr:firewalld.service is not enabled 2024-01-15T22:26:48.938 INFO:teuthology.orchestra.run.smithi099.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-01-15T22:26:48.940 INFO:teuthology.orchestra.run.smithi099.stderr:Ceph Dashboard is now available at: 2024-01-15T22:26:48.940 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-01-15T22:26:48.941 INFO:teuthology.orchestra.run.smithi099.stderr: URL: https://smithi099.front.sepia.ceph.com:8443/ 2024-01-15T22:26:48.941 INFO:teuthology.orchestra.run.smithi099.stderr: User: admin 2024-01-15T22:26:48.941 INFO:teuthology.orchestra.run.smithi099.stderr: Password: hyqb20d8sz 2024-01-15T22:26:48.941 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-01-15T22:26:48.941 INFO:teuthology.orchestra.run.smithi099.stderr:Enabling autotune for osd_memory_target 2024-01-15T22:26:49.179 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:49 smithi099 conmon[35292]: audit 2024-01-15T22:26:48.399632+0000 mon.smithi099 ( 2024-01-15T22:26:49.179 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:49 smithi099 conmon[35292]: mon.0) 91 : audit [DBG] from='client.? 172.21.15.99:0/4057207892' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-01-15T22:26:50.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:50 smithi099 conmon[35292]: cluster 2024-01-15T22:26:49.104039+0000 mon.smithi099 (mon.0) 92 : cluster [DBG] mgrmap e13: smithi099.cmjeyn(active, since 6s) 2024-01-15T22:26:50.741 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-01-15T22:26:51.288 INFO:teuthology.orchestra.run.smithi099.stderr:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-01-15T22:26:51.288 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-01-15T22:26:51.288 INFO:teuthology.orchestra.run.smithi099.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-01-15T22:26:51.288 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-01-15T22:26:51.288 INFO:teuthology.orchestra.run.smithi099.stderr:Or, if you are only running a single cluster on this host: 2024-01-15T22:26:51.288 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-01-15T22:26:51.289 INFO:teuthology.orchestra.run.smithi099.stderr: sudo /home/ubuntu/cephtest/cephadm shell 2024-01-15T22:26:51.289 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-01-15T22:26:51.289 INFO:teuthology.orchestra.run.smithi099.stderr:Please consider enabling telemetry to help improve Ceph: 2024-01-15T22:26:51.289 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-01-15T22:26:51.289 INFO:teuthology.orchestra.run.smithi099.stderr: ceph telemetry on 2024-01-15T22:26:51.289 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-01-15T22:26:51.289 INFO:teuthology.orchestra.run.smithi099.stderr:For more information see: 2024-01-15T22:26:51.289 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-01-15T22:26:51.289 INFO:teuthology.orchestra.run.smithi099.stderr: https://docs.ceph.com/en/pacific/mgr/telemetry/ 2024-01-15T22:26:51.289 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-01-15T22:26:51.289 INFO:teuthology.orchestra.run.smithi099.stderr:Bootstrap complete. 2024-01-15T22:26:51.336 INFO:tasks.cephadm:Fetching config... 2024-01-15T22:26:51.337 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-01-15T22:26:51.337 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-01-15T22:26:51.355 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-01-15T22:26:51.355 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-01-15T22:26:51.355 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-01-15T22:26:51.414 INFO:tasks.cephadm:Fetching mon keyring... 2024-01-15T22:26:51.414 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-01-15T22:26:51.414 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi099/keyring of=/dev/stdout 2024-01-15T22:26:51.483 INFO:tasks.cephadm:Fetching pub ssh key... 2024-01-15T22:26:51.483 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-01-15T22:26:51.483 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-01-15T22:26:51.541 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-01-15T22:26:51.541 DEBUG:teuthology.orchestra.run.smithi099:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCjTI9efurZE+QaaqGWYJLXbARGE72hpNNsu5b3cjdlHQdaDrW8EVlM2erTrkiSyh81ULv9XKu5LmuifrZnU5uCcrFr/GFTq5q0BXP+OToB89dz6xKnmdm1Vdi+LZlV7Xdi6+2mFA5n3xT7r6zIpyCTp+Yy33pjbcZT0379Zr//I0vyu3gG0cZtAE/KyhvyHIFTCliWtLw3378w/T72mwnzHaTC8N8hqkKYWeeGpfmlhgkkZizX16iqjJJd956/1Q6kkFxh6Xje5xvmOlY3nFWIabecbZEvjouWXSjIJ2m6Mt+KjBkGpYOBzzpf3GrIM2HXF0djaGh9/f1gXgSC/aQyKo6v8Fne1zN6I+QSnfxN3auRYMDnaygvUFDy4uc4Xu9IK6kVqUmkyWznsAr5PvQYEaxHC/2f8+kmItQK73Ws8B8hTI2x6HOugIGn8EgHc3K1pzswX6Xsu7ftxDLnisENGI+7/W0CUyeppN4jphwFYmDQKoLA3eFVNKXrEpJGVXE= ceph-eb17a8d4-b3f4-11ee-95ad-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-15T22:26:51.624 INFO:teuthology.orchestra.run.smithi099.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCjTI9efurZE+QaaqGWYJLXbARGE72hpNNsu5b3cjdlHQdaDrW8EVlM2erTrkiSyh81ULv9XKu5LmuifrZnU5uCcrFr/GFTq5q0BXP+OToB89dz6xKnmdm1Vdi+LZlV7Xdi6+2mFA5n3xT7r6zIpyCTp+Yy33pjbcZT0379Zr//I0vyu3gG0cZtAE/KyhvyHIFTCliWtLw3378w/T72mwnzHaTC8N8hqkKYWeeGpfmlhgkkZizX16iqjJJd956/1Q6kkFxh6Xje5xvmOlY3nFWIabecbZEvjouWXSjIJ2m6Mt+KjBkGpYOBzzpf3GrIM2HXF0djaGh9/f1gXgSC/aQyKo6v8Fne1zN6I+QSnfxN3auRYMDnaygvUFDy4uc4Xu9IK6kVqUmkyWznsAr5PvQYEaxHC/2f8+kmItQK73Ws8B8hTI2x6HOugIGn8EgHc3K1pzswX6Xsu7ftxDLnisENGI+7/W0CUyeppN4jphwFYmDQKoLA3eFVNKXrEpJGVXE= ceph-eb17a8d4-b3f4-11ee-95ad-87774f69a715 2024-01-15T22:26:51.636 DEBUG:teuthology.orchestra.run.smithi162:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCjTI9efurZE+QaaqGWYJLXbARGE72hpNNsu5b3cjdlHQdaDrW8EVlM2erTrkiSyh81ULv9XKu5LmuifrZnU5uCcrFr/GFTq5q0BXP+OToB89dz6xKnmdm1Vdi+LZlV7Xdi6+2mFA5n3xT7r6zIpyCTp+Yy33pjbcZT0379Zr//I0vyu3gG0cZtAE/KyhvyHIFTCliWtLw3378w/T72mwnzHaTC8N8hqkKYWeeGpfmlhgkkZizX16iqjJJd956/1Q6kkFxh6Xje5xvmOlY3nFWIabecbZEvjouWXSjIJ2m6Mt+KjBkGpYOBzzpf3GrIM2HXF0djaGh9/f1gXgSC/aQyKo6v8Fne1zN6I+QSnfxN3auRYMDnaygvUFDy4uc4Xu9IK6kVqUmkyWznsAr5PvQYEaxHC/2f8+kmItQK73Ws8B8hTI2x6HOugIGn8EgHc3K1pzswX6Xsu7ftxDLnisENGI+7/W0CUyeppN4jphwFYmDQKoLA3eFVNKXrEpJGVXE= ceph-eb17a8d4-b3f4-11ee-95ad-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-15T22:26:51.695 INFO:teuthology.orchestra.run.smithi162.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCjTI9efurZE+QaaqGWYJLXbARGE72hpNNsu5b3cjdlHQdaDrW8EVlM2erTrkiSyh81ULv9XKu5LmuifrZnU5uCcrFr/GFTq5q0BXP+OToB89dz6xKnmdm1Vdi+LZlV7Xdi6+2mFA5n3xT7r6zIpyCTp+Yy33pjbcZT0379Zr//I0vyu3gG0cZtAE/KyhvyHIFTCliWtLw3378w/T72mwnzHaTC8N8hqkKYWeeGpfmlhgkkZizX16iqjJJd956/1Q6kkFxh6Xje5xvmOlY3nFWIabecbZEvjouWXSjIJ2m6Mt+KjBkGpYOBzzpf3GrIM2HXF0djaGh9/f1gXgSC/aQyKo6v8Fne1zN6I+QSnfxN3auRYMDnaygvUFDy4uc4Xu9IK6kVqUmkyWznsAr5PvQYEaxHC/2f8+kmItQK73Ws8B8hTI2x6HOugIGn8EgHc3K1pzswX6Xsu7ftxDLnisENGI+7/W0CUyeppN4jphwFYmDQKoLA3eFVNKXrEpJGVXE= ceph-eb17a8d4-b3f4-11ee-95ad-87774f69a715 2024-01-15T22:26:51.707 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-01-15T22:26:51.903 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:51 smithi099 conmon[35292]: audit 2024-01-15T22:26:50.739417+0000 mon.smithi099 (mon.0) 93 : audit [INF] from='client.? 172.21.15.99:0/1310820994' entity='client.admin' 2024-01-15T22:26:54.660 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-01-15T22:26:54.661 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-01-15T22:26:55.273 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:55 smithi099 conmon[35292]: audit 2024-01-15T22:26:54.012835+0000 mon.smithi099 (mon.0) 94 : audit [INF] from='client.? 172.21.15.99:0/1738814655' entity='client.admin' 2024-01-15T22:26:56.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:56 smithi099 conmon[35292]: audit 2024-01-15T22:26:55.533323+0000 mon.smithi099 (mon.0) 95 : audit [INF] from='mgr.14156 172.21.15.99:0/3616479895' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:26:56.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:56 smithi099 conmon[35292]: audit 2024-01-15T22:26:56.065095+0000 mon.smithi099 (mon.0) 96 : audit [INF] from='mgr.14156 172.21.15.99:0/3616479895' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config rm", "who": "osd/host:smithi099", "name": "osd_memory_target"}]: dispatch 2024-01-15T22:26:56.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:56 smithi099 conmon[35292]: audit 2024-01-15T22:26:56.068172+0000 mon.smithi099 (mon.0) 97 : audit [INF] from='mgr.14156 172.21.15.99:0/3616479895' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:26:57.391 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi162 2024-01-15T22:26:57.392 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-01-15T22:26:57.392 DEBUG:teuthology.orchestra.run.smithi162:> dd of=/etc/ceph/ceph.conf 2024-01-15T22:26:57.413 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-01-15T22:26:57.413 DEBUG:teuthology.orchestra.run.smithi162:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-15T22:26:57.472 INFO:tasks.cephadm:Adding host smithi162 to orchestrator... 2024-01-15T22:26:57.472 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph orch host add smithi162 2024-01-15T22:26:57.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:57 smithi099 conmon[35292]: cephadm 2024-01-15T22:26:56.076028+0000 mgr.smithi099.cmjeyn (mgr.14156) 2024-01-15T22:26:57.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:57 smithi099 conmon[35292]: 5 : cephadm [INF] Deploying daemon alertmanager.smithi099 on smithi099 2024-01-15T22:26:57.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:57 smithi099 conmon[35292]: audit 2024-01-15T22:26:56.748990+0000 mon.smithi099 (mon.0) 98 : audit [INF] from='mgr.14156 172.21.15.99:0/3616479895' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:26:58.903 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:58 smithi099 conmon[35292]: audit 2024-01-15T22:26:56.746533+0000 mgr.smithi099.cmjeyn ( 2024-01-15T22:26:58.903 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:26:58 smithi099 conmon[35292]: mgr.14156) 6 : audit [DBG] from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-01-15T22:27:04.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:04 smithi099 conmon[35292]: cluster 2024-01-15T22:27:03.064477+0000 mgr.smithi099.cmjeyn (mgr.14156) 7 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:27:06.153 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:05 smithi099 conmon[35292]: cluster 2024-01-15T22:27:05.064912+0000 mgr.smithi099.cmjeyn (mgr.14156) 8 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:27:08.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:08 smithi099 conmon[35292]: cluster 2024-01-15T22:27:07.065137+0000 mgr.smithi099.cmjeyn (mgr.14156) 9 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:27:08.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:08 smithi099 conmon[35292]: 2024-01-15T22:27:10.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:10 smithi099 conmon[35292]: cluster 2024-01-15T22:27:09.065379+0000 mgr.smithi099.cmjeyn (mgr.14156) 10 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:27:12.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:12 smithi099 conmon[35292]: cluster 2024-01-15T22:27:11.065661+0000 mgr.smithi099.cmjeyn (mgr.14156) 11 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:27:13.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:13 smithi099 conmon[35292]: audit 2024-01-15T22:27:12.243886+0000 mon.smithi099 (mon.0) 99 : audit [INF] from='mgr.14156 172.21.15.99:0/3616479895' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:27:13.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:13 smithi099 conmon[35292]: audit 2024-01-15T22:27:12.245462+0000 mon.smithi099 (mon.0) 100 : audit [INF] from='mgr.14156 172.21.15.99:0/3616479895' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi099", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-15T22:27:13.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:13 smithi099 conmon[35292]: audit 2024-01-15T22:27:12.250386+0000 mon.smithi099 (mon.0) 101 : audit [INF] from='mgr.14156 172.21.15.99:0/3616479895' entity='mgr.smithi099.cmjeyn' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi099", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-01-15T22:27:13.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:13 smithi099 conmon[35292]: audit 2024-01-15T22:27:12.250976+0000 mon.smithi099 (mon.0) 102 : audit [DBG] from='mgr.14156 172.21.15.99:0/3616479895' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:27:13.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:13 smithi099 conmon[35292]: cephadm 2024-01-15T22:27:12.251438+0000 mgr.smithi099.cmjeyn (mgr.14156) 12 : cephadm [INF] Deploying daemon crash.smithi099 on smithi099 2024-01-15T22:27:13.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:13 smithi099 conmon[35292]: audit 2024-01-15T22:27:13.199220+0000 mon.smithi099 (mon.0) 103 : audit [INF] from='mgr.14156 172.21.15.99:0/3616479895' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:27:14.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:14 smithi099 conmon[35292]: cluster 2024-01-15T22:27:13.065937+0000 mgr.smithi099.cmjeyn ( 2024-01-15T22:27:14.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:14 smithi099 conmon[35292]: mgr.14156) 13 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:27:15.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:15 smithi099 conmon[35292]: audit 2024-01-15T22:27:14.352839+0000 mon.smithi099 (mon.0) 104 : audit 2024-01-15T22:27:15.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:15 smithi099 conmon[35292]: [INF] from='mgr.14156 172.21.15.99:0/3616479895' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:27:15.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:15 smithi099 conmon[35292]: audit 2024-01-15T22:27:14.461311+0000 mon.smithi099 (mon.0) 105 : audit [INF] from='mgr.14156 172.21.15.99:0/3616479895' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:27:15.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:15 smithi099 conmon[35292]: audit 2024-01-15T22:27:14.463189+0000 mon.smithi099 (mon.0) 106 : audit [INF] from='mgr.14156 172.21.15.99:0/3616479895' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:27:15.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:15 smithi099 conmon[35292]: audit 2024-01-15T22:27:14.465203+0000 mon.smithi099 (mon.0) 107 : audit [INF] from='mgr.14156 172.21.15.99:0/3616479895' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-15T22:27:15.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:15 smithi099 conmon[35292]: audit 2024-01-15T22:27:14.465539+0000 mgr.smithi099.cmjeyn (mgr.14156) 14 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-15T22:27:15.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:15 smithi099 conmon[35292]: audit 2024-01-15 2024-01-15T22:27:15.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:15 smithi099 conmon[35292]: T22:27:14.467977+0000 mon.smithi099 (mon.0) 108 : audit [INF] from='mgr.14156 172.21.15.99:0/3616479895' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:27:15.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:15 smithi099 conmon[35292]: cephadm 2024-01-15 2024-01-15T22:27:15.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:15 smithi099 conmon[35292]: T22:27:14.474817+0000 mgr.smithi099.cmjeyn (mgr. 2024-01-15T22:27:15.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:15 smithi099 conmon[35292]: 14156) 15 : cephadm [INF] Deploying daemon grafana.smithi099 on smithi099 2024-01-15T22:27:16.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:16 smithi099 conmon[35292]: cluster 2024-01-15T22:27:15.066171+0000 mgr.smithi099.cmjeyn (mgr.14156) 16 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:27:18.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:18 smithi099 conmon[35292]: cluster 2024-01-15T22:27:17.066396+0000 2024-01-15T22:27:18.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:18 smithi099 conmon[35292]: mgr.smithi099.cmjeyn (mgr.14156) 17 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:27:18.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:18 smithi099 conmon[35292]: audit 2024-01-15T22:27:18.206948+0000 mon.smithi099 (mon 2024-01-15T22:27:18.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:18 smithi099 conmon[35292]: .0) 109 : audit [INF] from='mgr.14156 172.21.15.99:0/3616479895' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:27:20.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:20 smithi099 conmon[35292]: cluster 2024-01-15T22:27:19.066613+0000 mgr.smithi099.cmjeyn (mgr.14156) 18 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:27:22.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:22 smithi099 conmon[35292]: cluster 2024-01-15T22:27:21.066850+0000 mgr.smithi099.cmjeyn (mgr.14156) 19 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:27:24.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:24 smithi099 conmon[35292]: cluster 2024-01-15T22:27:23.067134+0000 2024-01-15T22:27:24.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:24 smithi099 conmon[35292]: mgr.smithi099.cmjeyn (mgr.14156) 20 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:27:26.153 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:25 smithi099 conmon[35292]: cluster 2024-01-15T22:27:25.067394+0000 mgr.smithi099.cmjeyn (mgr.14156) 2024-01-15T22:27:26.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:25 smithi099 conmon[35292]: 21 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:27:28.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:28 smithi099 conmon[35292]: cluster 2024-01-15T22:27:27.067741+0000 mgr.smithi099.cmjeyn (mgr.14156) 22 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:27:30.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:30 smithi099 conmon[35292]: cluster 2024-01-15T22:27:29.068152+0000 mgr.smithi099.cmjeyn (mgr.14156) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:27:32.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:32 smithi099 conmon[35292]: cluster 2024-01-15T22:27:31.068497+0000 mgr.smithi099.cmjeyn (mgr.14156) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:27:34.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:34 smithi099 conmon[35292]: cluster 2024-01-15T22:27:33.068872+0000 mgr.smithi099.cmjeyn (mgr.14156 2024-01-15T22:27:34.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:34 smithi099 conmon[35292]: ) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:27:36.310 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:36 smithi099 conmon[35292]: cluster 2024-01-15T22:27:35.069234+0000 mgr.smithi099.cmjeyn (mgr. 2024-01-15T22:27:36.311 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:36 smithi099 conmon[35292]: 14156) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:27:36.311 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:36 smithi099 conmon[35292]: audit 2024-01-15T22:27:35.287645+0000 mon.smithi099 (mon.0) 110 : audit [INF] from='mgr.14156 172.21.15.99:0/3616479895' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:27:36.311 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:36 smithi099 conmon[35292]: 2024-01-15T22:27:36.311 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:36 smithi099 conmon[35292]: cephadm 2024-01-15T22:27:35 2024-01-15T22:27:36.311 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:36 smithi099 conmon[35292]: .289621+0000 mgr.smithi099.cmjeyn (mgr.14156) 27 : cephadm [INF] Deploying daemon node-exporter.smithi099 on smithi099 2024-01-15T22:27:36.627 INFO:teuthology.orchestra.run.smithi162.stdout:Added host 'smithi162' with addr '172.21.15.162' 2024-01-15T22:27:37.139 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph orch host ls --format=json 2024-01-15T22:27:37.687 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:37 smithi099 conmon[35292]: audit 2024-01-15T22:27:36.005196+0000 mgr.smithi099.cmjeyn (mgr.14156) 28 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi162", "target": ["mon-mgr", ""]}]: dispatch 2024-01-15T22:27:37.687 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:37 smithi099 conmon[35292]: audit 2024-01-15T22: 2024-01-15T22:27:37.687 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:37 smithi099 conmon[35292]: 27:36.626962+0000 mon.smithi099 (mon.0) 111 : audit [INF] from='mgr.14156 172.21.15.99:0/3616479895' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:27:37.687 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:37 smithi099 conmon[35292]: cephadm 2024-01-15T22:27:36.627240+0000 mgr.smithi099.cmjeyn (mgr.14156) 29 : cephadm [INF] 2024-01-15T22:27:37.687 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:37 smithi099 conmon[35292]: Added host smithi162 2024-01-15T22:27:38.735 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-15T22:27:38.735 INFO:teuthology.orchestra.run.smithi162.stdout:[{"addr": "172.21.15.99", "hostname": "smithi099", "labels": [], "status": ""}, {"addr": "172.21.15.162", "hostname": "smithi162", "labels": [], "status": ""}] 2024-01-15T22:27:38.857 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:38 smithi099 conmon[35292]: cluster 2024-01-15T22:27:37.069810+0000 mgr.smithi099.cmjeyn (mgr.14156 2024-01-15T22:27:38.857 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:38 smithi099 conmon[35292]: ) 30 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:27:38.857 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:38 smithi099 conmon[35292]: audit 2024-01-15T22:27:38.211511+0000 mon.smithi099 (mon 2024-01-15T22:27:38.857 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:38 smithi099 conmon[35292]: .0) 112 : audit [INF] from='mgr.14156 172.21.15.99:0/3616479895' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:27:39.254 INFO:tasks.cephadm:Setting crush tunables to default 2024-01-15T22:27:39.255 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd crush tunables default 2024-01-15T22:27:39.901 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:39 smithi099 conmon[35292]: audit 2024-01-15T22:27:38.734054+0000 mgr.smithi099.cmjeyn (mgr.14156) 2024-01-15T22:27:39.901 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:39 smithi099 conmon[35292]: 31 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-15T22:27:39.901 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:39 smithi099 conmon[35292]: audit 2024-01-15T22:27:38.736872+0000 mon.smithi099 (mon.0) 113 : audit [INF] from='mgr.14156 172.21.15.99:0/3616479895' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:27:39.901 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:39 smithi099 conmon[35292]: audit 2024-01-15 2024-01-15T22:27:39.902 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:39 smithi099 conmon[35292]: T22:27:38.739201+0000 mon.smithi099 (mon.0) 114 : audit [INF] from='mgr.14156 172.21.15.99:0/3616479895' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-01-15T22:27:39.902 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:39 smithi099 conmon[35292]: cluster 2024-01-15T22:27:39.070017+0000 2024-01-15T22:27:39.902 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:39 smithi099 conmon[35292]: mgr.smithi099.cmjeyn (mgr.14156) 32 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:27:40.896 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:40 smithi099 conmon[35292]: audit 2024-01-15T22:27:39.739910+0000 mon.smithi099 (mon.0) 115 : audit [INF] from='mgr.14156 172.21.15.99:0/3616479895' entity='mgr.smithi099.cmjeyn' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-01-15T22:27:40.896 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:40 smithi099 conmon[35292]: cluster 2024-01-15T22:27:39.739987+0000 2024-01-15T22:27:40.896 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:40 smithi099 conmon[35292]: mon.smithi099 (mon.0) 116 : cluster [DBG] mgrmap e14: smithi099.cmjeyn(active, since 56s) 2024-01-15T22:27:41.856 INFO:teuthology.orchestra.run.smithi099.stderr:adjusted tunables profile to default 2024-01-15T22:27:41.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:41 smithi099 conmon[35292]: audit 2024-01-15T22:27:41.799034+0000 mon.smithi099 (mon.0) 117 : 2024-01-15T22:27:41.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:41 smithi099 conmon[35292]: audit [INF] from='client.? 172.21.15.99:0/4248350280' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-01-15T22:27:42.637 INFO:tasks.cephadm:Adding mon.smithi099 on smithi099 2024-01-15T22:27:42.637 INFO:tasks.cephadm:Adding mon.smithi162 on smithi162 2024-01-15T22:27:42.637 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph orch apply mon '2;smithi099:172.21.15.99=smithi099;smithi162:172.21.15.162=smithi162' 2024-01-15T22:27:42.878 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:42 smithi099 conmon[35292]: audit 2024-01-15T22:27:41.853636+0000 mon.smithi099 (mon.0) 118 : audit [INF] from='client.? 172.21.15.99:0/4248350280' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-01-15T22:27:42.878 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:42 smithi099 conmon[35292]: cluster 2024-01-15T22:27:41.853696+0000 2024-01-15T22:27:42.878 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:42 smithi099 conmon[35292]: mon.smithi099 (mon.0) 119 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-01-15T22:27:46.131 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:45 smithi099 conmon[35292]: cluster 2024-01-15T22:27:45.196713+0000 mon.smithi099 (mon.0) 120 : cluster [INF] 2024-01-15T22:27:46.131 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:45 smithi099 conmon[35292]: Active manager daemon smithi099.cmjeyn restarted 2024-01-15T22:27:46.131 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:45 smithi099 conmon[35292]: cluster 2024-01-15T22:27:45.196971+0000 mon.smithi099 (mon.0) 121 : cluster [INF] Activating manager daemon smithi099.cmjeyn 2024-01-15T22:27:46.132 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:45 smithi099 conmon[35292]: cluster 2024-01-15T22:27:45.199830+0000 mon.smithi099 (mon.0) 122 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-01-15T22:27:46.132 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:45 smithi099 conmon[35292]: cluster 2024-01-15T22:27:45.199936+0000 mon.smithi099 (mon.0) 123 : cluster [DBG] mgrmap e15: smithi099.cmjeyn(active, starting, since 0.00305819s) 2024-01-15T22:27:46.132 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:45 smithi099 conmon[35292]: audit 2024-01-15T22:27:45.200747+0000 mon.smithi099 (mon.0) 124 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-01-15T22:27:46.132 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:45 smithi099 conmon[35292]: audit 2024-01-15T22:27:45.201341+0000 mon.smithi099 (mon.0) 125 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mgr metadata", "who": "smithi099.cmjeyn", "id": "smithi099.cmjeyn"}]: dispatch 2024-01-15T22:27:46.132 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:45 smithi099 conmon[35292]: audit 2024-01-15T22:27:45.202138+0000 mon.smithi099 (mon.0) 126 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-15T22:27:46.132 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:45 smithi099 conmon[35292]: audit 2024-01-15T22:27:45.202300+0000 mon.smithi099 (mon.0) 127 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-15T22:27:46.132 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:45 smithi099 conmon[35292]: audit 2024-01-15T22 2024-01-15T22:27:46.132 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:45 smithi099 conmon[35292]: :27:45.202432+0000 mon.smithi099 (mon.0) 128 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-15T22:27:46.132 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:45 smithi099 conmon[35292]: cluster 2024-01- 2024-01-15T22:27:46.132 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:45 smithi099 conmon[35292]: 15T22:27:45.216379+0000 mon.smithi099 (mon.0) 2024-01-15T22:27:46.132 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:45 smithi099 conmon[35292]: 129 : cluster [INF] Manager daemon smithi099.cmjeyn is now available 2024-01-15T22:27:46.132 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:45 smithi099 conmon[35292]: audit 2024-01-15T22:27:45.314189+0000 2024-01-15T22:27:46.132 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:45 smithi099 conmon[35292]: mon.smithi099 (mon.0) 130 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-15T22:27:46.133 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:45 smithi099 conmon[35292]: audit 2024-01-15T22:27:45. 2024-01-15T22:27:46.133 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:45 smithi099 conmon[35292]: 315645+0000 mon.smithi099 (mon.0) 131 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-15T22:27:46.133 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:45 smithi099 conmon[35292]: audit 2024-01-15T22:27:45.344907+0000 mon.smithi099 (mon.0) 132 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:27:46.133 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:45 smithi099 conmon[35292]: audit 2024-01-15T22: 2024-01-15T22:27:46.133 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:45 smithi099 conmon[35292]: 27:45.358917+0000 mon.smithi099 (mon.0) 133 : audit 2024-01-15T22:27:46.133 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:45 smithi099 conmon[35292]: [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi099.cmjeyn/mirror_snapshot_schedule"}]: dispatch 2024-01-15T22:27:46.133 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:45 smithi099 conmon[35292]: audit 2024-01-15T22:27:45 2024-01-15T22:27:46.133 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:45 smithi099 conmon[35292]: .361897+0000 mon.smithi099 (mon.0) 134 : audit 2024-01-15T22:27:46.133 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:45 smithi099 conmon[35292]: [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi099.cmjeyn/trash_purge_schedule"}]: dispatch 2024-01-15T22:27:46.133 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:45 smithi099 conmon[35292]: audit 2024-01-15T22:27:45.804123 2024-01-15T22:27:46.133 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:45 smithi099 conmon[35292]: +0000 mon.smithi099 (mon.0) 135 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:27:46.208 INFO:teuthology.orchestra.run.smithi162.stdout:Scheduled mon update... 2024-01-15T22:27:46.745 DEBUG:teuthology.orchestra.run.smithi162:mon.smithi162> sudo journalctl -f -n 0 -u ceph-eb17a8d4-b3f4-11ee-95ad-87774f69a715@mon.smithi162.service 2024-01-15T22:27:46.747 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-15T22:27:46.747 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph mon dump -f json 2024-01-15T22:27:46.779 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:-- Logs begin at Mon 2024-01-15 22:13:05 UTC. -- 2024-01-15T22:27:47.261 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:47 smithi099 conmon[35292]: audit 2024-01-15T22:27:46.107241+0000 mon.smithi099 (mon.0) 136 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:27:47.262 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:47 smithi099 conmon[35292]: cluster 2024-01-15T22:27:46.204196+0000 mon.smithi099 (mon.0) 137 : cluster [DBG] mgrmap e16: smithi099.cmjeyn(active, since 1.00732s) 2024-01-15T22:27:47.262 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:47 smithi099 conmon[35292]: cephadm 2024-01-15T 2024-01-15T22:27:47.262 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:47 smithi099 conmon[35292]: 22:27:46.206323+0000 mgr.smithi099.cmjeyn (mgr.14184) 2 2024-01-15T22:27:47.262 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:47 smithi099 conmon[35292]: : cephadm [INF] Saving service mon spec with placement smithi099:172.21.15.99=smithi099;smithi162:172.21.15.162=smithi162;count:2 2024-01-15T22:27:47.262 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:47 smithi099 conmon[35292]: audit 2024-01-15T22:27:46 2024-01-15T22:27:47.262 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:47 smithi099 conmon[35292]: .208347+0000 mon.smithi099 (mon.0) 138 : 2024-01-15T22:27:47.262 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:47 smithi099 conmon[35292]: audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:27:48.971 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-15T22:27:48.971 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":1,"fsid":"eb17a8d4-b3f4-11ee-95ad-87774f69a715","modified":"2024-01-15T22:25:52.892888Z","created":"2024-01-15T22:25:52.892888Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi099","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:3300","nonce":0},{"type":"v1","addr":"172.21.15.99:6789","nonce":0}]},"addr":"172.21.15.99:6789/0","public_addr":"172.21.15.99:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-15T22:27:48.973 INFO:teuthology.orchestra.run.smithi162.stderr:dumped monmap epoch 1 2024-01-15T22:27:49.150 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:49 smithi099 conmon[35292]: cluster 2024-01-15T22:27:48.109308+0000 mon.smithi099 (mon.0) 139 : cluster [DBG] mgrmap e17: smithi099.cmjeyn(active, since 2s) 2024-01-15T22:27:49.150 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:49 smithi099 conmon[35292]: audit 2024-01-15T22:27:48.355716+0000 mon.smithi099 (mon.0) 140 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:27:49.150 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:49 smithi099 conmon[35292]: audit 2024-01-15T22:27:49.150 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:49 smithi099 conmon[35292]: 2024-01-15T22:27:48.664261+0000 mon.smithi099 (mon.0) 141 : audit [INF] 2024-01-15T22:27:49.150 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:49 smithi099 conmon[35292]: from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config rm", "who": "osd/host:smithi099", "name": "osd_memory_target"}]: dispatch 2024-01-15T22:27:49.150 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:49 smithi099 conmon[35292]: audit 2024-01-15T22:27 2024-01-15T22:27:49.150 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:49 smithi099 conmon[35292]: :48.971233+0000 mon.smithi099 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.162:0/1118191476' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-15T22:27:50.663 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-15T22:27:50.664 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph mon dump -f json 2024-01-15T22:27:51.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:51 smithi099 conmon[35292]: audit 2024-01-15T22:27:50. 2024-01-15T22:27:51.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:51 smithi099 conmon[35292]: 324347+0000 mon.smithi099 (mon.0) 143 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:27:51.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:51 smithi099 conmon[35292]: audit 2024-01-15T22:27:51.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:51 smithi099 conmon[35292]: 2024-01-15T22:27:50.818578+0000 mon.smithi099 (mon.0) 144 : audit [INF] 2024-01-15T22:27:51.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:51 smithi099 conmon[35292]: from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:27:51.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:51 smithi099 conmon[35292]: audit 2024-01-15T22:27:50. 2024-01-15T22:27:51.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:51 smithi099 conmon[35292]: 819448+0000 mon.smithi099 (mon.0) 145 : audit [INF] 2024-01-15T22:27:51.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:51 smithi099 conmon[35292]: from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config rm", "who": "osd/host:smithi162", "name": "osd_memory_target"}]: dispatch 2024-01-15T22:27:51.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:51 smithi099 conmon[35292]: audit 2024-01-15T22:27:50.820603+0000 2024-01-15T22:27:51.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:51 smithi099 conmon[35292]: mon.smithi099 (mon.0) 146 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:27:51.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:51 smithi099 conmon[35292]: audit 2024- 2024-01-15T22:27:51.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:51 smithi099 conmon[35292]: 01-15T22:27:50.821453+0000 mon.smithi099 ( 2024-01-15T22:27:51.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:51 smithi099 conmon[35292]: mon.0) 147 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-15T22:27:51.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:51 smithi099 conmon[35292]: audit 2024-01-15T22:27:51.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:51 smithi099 conmon[35292]: 2024-01-15T22:27:50 2024-01-15T22:27:51.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:51 smithi099 conmon[35292]: .893555+0000 mon.smithi099 (mon.0) 148 : audit [INF] 2024-01-15T22:27:51.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:51 smithi099 conmon[35292]: from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:27:51.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:51 smithi099 conmon[35292]: audit 2024-01-15T22:27: 2024-01-15T22:27:51.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:51 smithi099 conmon[35292]: 50.910281+0000 mon.smithi099 (mon.0) 149 : audit 2024-01-15T22:27:51.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:51 smithi099 conmon[35292]: [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:27:51.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:51 smithi099 conmon[35292]: audit 2024- 2024-01-15T22:27:51.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:51 smithi099 conmon[35292]: 01-15T22:27:50.913312+0000 mon.smithi099 (mon 2024-01-15T22:27:51.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:51 smithi099 conmon[35292]: .0) 150 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:27:51.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:51 smithi099 conmon[35292]: audit 2024-01-15T22:27: 2024-01-15T22:27:51.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:51 smithi099 conmon[35292]: 50.915080+0000 mon.smithi099 (mon.0) 151 : audit 2024-01-15T22:27:51.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:51 smithi099 conmon[35292]: [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi162", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-15T22:27:51.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:51 smithi099 conmon[35292]: audit 2024-01-15T22:27:50.917151 2024-01-15T22:27:51.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:51 smithi099 conmon[35292]: +0000 mon.smithi099 (mon.0) 152 : audit [INF] 2024-01-15T22:27:51.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:51 smithi099 conmon[35292]: from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi162", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-01-15T22:27:51.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:51 smithi099 conmon[35292]: audit 2024-01-15T22:27:51.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:51 smithi099 conmon[35292]: 2024-01-15T22:27:50.917959+0000 mon.smithi099 (mon.0) 2024-01-15T22:27:51.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:51 smithi099 conmon[35292]: 153 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:27:52.577 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-15T22:27:52.578 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":1,"fsid":"eb17a8d4-b3f4-11ee-95ad-87774f69a715","modified":"2024-01-15T22:25:52.892888Z","created":"2024-01-15T22:25:52.892888Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi099","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:3300","nonce":0},{"type":"v1","addr":"172.21.15.99:6789","nonce":0}]},"addr":"172.21.15.99:6789/0","public_addr":"172.21.15.99:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-15T22:27:52.580 INFO:teuthology.orchestra.run.smithi162.stderr:dumped monmap epoch 1 2024-01-15T22:27:52.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:52 smithi099 conmon[35292]: cephadm 2024-01-15T22:27:50.822795+0000 mgr.smithi099.cmjeyn ( 2024-01-15T22:27:52.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:52 smithi099 conmon[35292]: mgr.14184) 3 : cephadm [INF] Updating smithi099:/etc/ceph/ceph.conf 2024-01-15T22:27:52.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:52 smithi099 conmon[35292]: cephadm 2024-01-15T22:27:50.823118+0000 mgr.smithi099.cmjeyn (mgr.14184) 4 : cephadm [INF] Updating smithi162:/etc/ceph/ceph.conf 2024-01-15T22:27:52.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:52 smithi099 conmon[35292]: cephadm 2024-01-15T22:27:50.839639+0000 mgr.smithi099.cmjeyn (mgr.14184) 5 : cephadm [INF] 2024-01-15T22:27:52.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:52 smithi099 conmon[35292]: Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-01-15T22:27:52.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:52 smithi099 conmon[35292]: cephadm 2024-01-15T22:27:50.865341+0000 mgr.smithi099.cmjeyn 2024-01-15T22:27:52.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:52 smithi099 conmon[35292]: (mgr.14184) 6 : cephadm [INF] Updating smithi099:/etc/ceph/ceph.client.admin.keyring 2024-01-15T22:27:52.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:52 smithi099 conmon[35292]: cephadm 2024-01-15T 2024-01-15T22:27:52.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:52 smithi099 conmon[35292]: 22:27:50.918632+0000 mgr.smithi099.cmjeyn (mgr.14184) 7 : cephadm 2024-01-15T22:27:52.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:52 smithi099 conmon[35292]: [INF] Deploying daemon crash.smithi162 on smithi162 2024-01-15T22:27:53.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:53 smithi099 conmon[35292]: audit 2024-01-15T22:27:52.578225+0000 mon.smithi099 (mon.0) 154 : 2024-01-15T22:27:53.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:53 smithi099 conmon[35292]: audit [DBG] from='client.? 172.21.15.162:0/1089194542' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-15T22:27:54.423 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-15T22:27:54.423 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph mon dump -f json 2024-01-15T22:27:55.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:55 smithi099 conmon[35292]: audit 2024-01-15T22:27:54.320718+0000 mon.smithi099 (mon.0) 155 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:27:55.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:55 smithi099 conmon[35292]: audit 2024-01-15T22:27:54.322854+0000 mon.smithi099 (mon.0) 156 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi162.vbobpt", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-15T22:27:55.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:55 smithi099 conmon[35292]: audit 2024-01-15T22:27:54.324828 2024-01-15T22:27:55.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:55 smithi099 conmon[35292]: +0000 mon.smithi099 (mon.0) 157 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi162.vbobpt", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-01-15T22:27:55.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:55 smithi099 conmon[35292]: audit 2024-01-15T22:27:54.325597+0000 2024-01-15T22:27:55.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:55 smithi099 conmon[35292]: mon.smithi099 (mon.0) 158 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-15T22:27:55.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:55 smithi099 conmon[35292]: audit 2024-01-15T22:27:54.326391+0000 mon.smithi099 2024-01-15T22:27:55.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:55 smithi099 conmon[35292]: (mon.0) 159 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:27:55.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:55 smithi099 conmon[35292]: cephadm 2024-01-15T 2024-01-15T22:27:55.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:55 smithi099 conmon[35292]: 22:27:54.327015+0000 mgr.smithi099.cmjeyn (mgr.14184) 8 : cephadm 2024-01-15T22:27:55.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:55 smithi099 conmon[35292]: [INF] Deploying daemon mgr.smithi162.vbobpt on smithi162 2024-01-15T22:27:56.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:56 smithi099 conmon[35292]: audit 2024-01-15T22:27:55.347345+0000 mon.smithi099 (mon.0) 160 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:27:56.945 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-15T22:27:56.945 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":1,"fsid":"eb17a8d4-b3f4-11ee-95ad-87774f69a715","modified":"2024-01-15T22:25:52.892888Z","created":"2024-01-15T22:25:52.892888Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi099","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:3300","nonce":0},{"type":"v1","addr":"172.21.15.99:6789","nonce":0}]},"addr":"172.21.15.99:6789/0","public_addr":"172.21.15.99:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-15T22:27:56.947 INFO:teuthology.orchestra.run.smithi162.stderr:dumped monmap epoch 1 2024-01-15T22:27:57.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:57 smithi099 conmon[35292]: audit 2024-01-15T22:27:56.945064+0000 mon.smithi099 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.162:0/3189128017' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-15T22:27:57.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:57 smithi099 conmon[35292]: audit 2024-01-15T22:27:57.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:57 smithi099 conmon[35292]: 2024-01-15T22:27:57.135650+0000 mon.smithi099 (mon.0) 162 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:27:57.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:57 smithi099 conmon[35292]: audit 2024-01-15T22:27:57.137729+0000 mon.smithi099 (mon.0) 163 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-15T22:27:57.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:57 smithi099 conmon[35292]: audit 2024-01-15T22:27:57.138503+0000 mon.smithi099 (mon.0) 164 : 2024-01-15T22:27:57.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:57 smithi099 conmon[35292]: audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:27:58.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:27:58 smithi099 conmon[35292]: cephadm 2024-01-15T22:27:57.139206+0000 mgr.smithi099.cmjeyn (mgr.14184) 9 : cephadm [INF] Deploying daemon mon.smithi162 on smithi162 2024-01-15T22:27:58.698 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-15T22:27:58.698 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph mon dump -f json 2024-01-15T22:28:00.517 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:00 smithi162 systemd[1]: Starting Ceph mon.smithi162 for eb17a8d4-b3f4-11ee-95ad-87774f69a715... 2024-01-15T22:28:00.958 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-15T22:28:00.959 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":1,"fsid":"eb17a8d4-b3f4-11ee-95ad-87774f69a715","modified":"2024-01-15T22:25:52.892888Z","created":"2024-01-15T22:25:52.892888Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi099","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:3300","nonce":0},{"type":"v1","addr":"172.21.15.99:6789","nonce":0}]},"addr":"172.21.15.99:6789/0","public_addr":"172.21.15.99:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-15T22:28:00.960 INFO:teuthology.orchestra.run.smithi162.stderr:dumped monmap epoch 1 2024-01-15T22:28:01.025 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:00 smithi162 conmon[39755]: debug 2024-01-15T22:28:00.998+0000 7fabb8e34880 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-15T22:28:01.025 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:00 smithi162 conmon[39755]: debug 2024-01-15T22:28:00.998+0000 7fabb8e34880 0 ceph version 16.2.14-456-g445562ab (445562ab4bc3ddfb386936119050695810860bcb) pacific (stable), process ceph-mon, pid 7 2024-01-15T22:28:01.025 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:00 smithi162 conmon[39755]: debug 2024-01-15T22:28:00.998+0000 7fabb8e34880 0 pidfile_write: ignore empty --pid-file 2024-01-15T22:28:01.026 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.002+0000 7fabb8e34880 0 load: jerasure load: lrc load: isa 2024-01-15T22:28:01.026 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: RocksDB version: 6.8.1 2024-01-15T22:28:01.026 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: 2024-01-15T22:28:01.026 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-15T22:28:01.026 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Compile date Jan 12 2024 2024-01-15T22:28:01.026 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: DB SUMMARY 2024-01-15T22:28:01.026 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: 2024-01-15T22:28:01.026 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: CURRENT file: CURRENT 2024-01-15T22:28:01.026 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: 2024-01-15T22:28:01.026 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: IDENTITY file: IDENTITY 2024-01-15T22:28:01.026 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: 2024-01-15T22:28:01.026 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: MANIFEST file: MANIFEST-000001 size: 13 Bytes 2024-01-15T22:28:01.026 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: 2024-01-15T22:28:01.027 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi162/store.db dir, Total Num: 0, files: 2024-01-15T22:28:01.027 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: 2024-01-15T22:28:01.027 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi162/store.db: 000003.log size: 511 ; 2024-01-15T22:28:01.027 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: 2024-01-15T22:28:01.027 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.error_if_exists: 0 2024-01-15T22:28:01.027 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.create_if_missing: 0 2024-01-15T22:28:01.027 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.paranoid_checks: 1 2024-01-15T22:28:01.027 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.env: 0x558f89cec080 2024-01-15T22:28:01.027 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.fs: Posix File System 2024-01-15T22:28:01.027 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.info_log: 0x558f8b50bd20 2024-01-15T22:28:01.027 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-15T22:28:01.027 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.statistics: (nil) 2024-01-15T22:28:01.027 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.use_fsync: 0 2024-01-15T22:28:01.027 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.max_log_file_size: 0 2024-01-15T22:28:01.028 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-15T22:28:01.028 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-15T22:28:01.028 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-15T22:28:01.028 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-15T22:28:01.028 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.allow_fallocate: 1 2024-01-15T22:28:01.028 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-15T22:28:01.028 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-15T22:28:01.028 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.use_direct_reads: 0 2024-01-15T22:28:01.028 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-15T22:28:01.028 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.create_missing_column_families: 0 2024-01-15T22:28:01.028 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.db_log_dir: 2024-01-15T22:28:01.028 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi162/store.db 2024-01-15T22:28:01.028 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-15T22:28:01.028 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.max_subcompactions: 1 2024-01-15T22:28:01.029 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.max_background_flushes: -1 2024-01-15T22:28:01.029 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-15T22:28:01.029 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-15T22:28:01.029 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-15T22:28:01.029 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-15T22:28:01.029 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-15T22:28:01.029 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.advise_random_on_open: 1 2024-01-15T22:28:01.029 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-15T22:28:01.029 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.write_buffer_manager: 0x558f8c162480 2024-01-15T22:28:01.029 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-15T22:28:01.029 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-15T22:28:01.029 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-15T22:28:01.029 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-15T22:28:01.029 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.rate_limiter: (nil) 2024-01-15T22:28:01.029 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-15T22:28:01.030 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-15T22:28:01.030 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-15T22:28:01.030 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-15T22:28:01.030 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.unordered_write: 0 2024-01-15T22:28:01.030 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-15T22:28:01.030 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-15T22:28:01.030 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-15T22:28:01.030 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-15T22:28:01.030 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.row_cache: None 2024-01-15T22:28:01.030 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.wal_filter: None 2024-01-15T22:28:01.030 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-15T22:28:01.030 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-15T22:28:01.030 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.preserve_deletes: 0 2024-01-15T22:28:01.030 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.two_write_queues: 0 2024-01-15T22:28:01.031 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.manual_wal_flush: 0 2024-01-15T22:28:01.031 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.atomic_flush: 0 2024-01-15T22:28:01.031 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-15T22:28:01.031 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-15T22:28:01.031 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-15T22:28:01.031 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.log_readahead_size: 0 2024-01-15T22:28:01.031 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-15T22:28:01.031 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.max_background_jobs: 2 2024-01-15T22:28:01.031 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.max_background_compactions: -1 2024-01-15T22:28:01.031 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-15T22:28:01.031 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-15T22:28:01.031 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-15T22:28:01.031 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.max_total_wal_size: 0 2024-01-15T22:28:01.031 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-15T22:28:01.031 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-15T22:28:01.032 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-15T22:28:01.032 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-15T22:28:01.032 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.max_open_files: -1 2024-01-15T22:28:01.032 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.bytes_per_sync: 0 2024-01-15T22:28:01.032 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-15T22:28:01.032 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-15T22:28:01.032 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-15T22:28:01.032 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Compression algorithms supported: 2024-01-15T22:28:01.032 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-15T22:28:01.032 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: kZSTD supported: 0 2024-01-15T22:28:01.032 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: kXpressCompression supported: 0 2024-01-15T22:28:01.032 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-15T22:28:01.032 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: kLZ4Compression supported: 1 2024-01-15T22:28:01.032 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: kBZip2Compression supported: 0 2024-01-15T22:28:01.032 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: kZlibCompression supported: 1 2024-01-15T22:28:01.033 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: kSnappyCompression supported: 1 2024-01-15T22:28:01.033 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-15T22:28:01.033 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi162/store.db/MANIFEST-000001 2024-01-15T22:28:01.033 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: 2024-01-15T22:28:01.033 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-15T22:28:01.033 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: 2024-01-15T22:28:01.033 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-15T22:28:01.033 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.merge_operator: 2024-01-15T22:28:01.033 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.compaction_filter: None 2024-01-15T22:28:01.033 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.compaction_filter_factory: None 2024-01-15T22:28:01.033 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-15T22:28:01.034 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-15T22:28:01.034 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558f8b4b5d28) 2024-01-15T22:28:01.034 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: cache_index_and_filter_blocks: 1 2024-01-15T22:28:01.034 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-15T22:28:01.034 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-15T22:28:01.034 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: pin_top_level_index_and_filter: 1 2024-01-15T22:28:01.034 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: index_type: 0 2024-01-15T22:28:01.035 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: data_block_index_type: 0 2024-01-15T22:28:01.035 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: index_shortening: 1 2024-01-15T22:28:01.035 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: data_block_hash_table_util_ratio: 0.750000 2024-01-15T22:28:01.035 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: hash_index_allow_collision: 1 2024-01-15T22:28:01.035 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: checksum: 1 2024-01-15T22:28:01.035 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: no_block_cache: 0 2024-01-15T22:28:01.035 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: block_cache: 0x558f8b4eef10 2024-01-15T22:28:01.035 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: block_cache_name: BinnedLRUCache 2024-01-15T22:28:01.035 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: block_cache_options: 2024-01-15T22:28:01.035 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: capacity : 536870912 2024-01-15T22:28:01.035 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: num_shard_bits : 4 2024-01-15T22:28:01.035 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: strict_capacity_limit : 0 2024-01-15T22:28:01.035 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: high_pri_pool_ratio: 0.000 2024-01-15T22:28:01.035 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: block_cache_compressed: (nil) 2024-01-15T22:28:01.036 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: persistent_cache: (nil) 2024-01-15T22:28:01.036 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: block_size: 4096 2024-01-15T22:28:01.036 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: block_size_deviation: 10 2024-01-15T22:28:01.036 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: block_restart_interval: 16 2024-01-15T22:28:01.036 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: index_block_restart_interval: 1 2024-01-15T22:28:01.036 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: metadata_block_size: 4096 2024-01-15T22:28:01.036 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: partition_filters: 0 2024-01-15T22:28:01.036 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: use_delta_encoding: 1 2024-01-15T22:28:01.036 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-15T22:28:01.036 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: whole_key_filtering: 1 2024-01-15T22:28:01.036 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: verify_compression: 0 2024-01-15T22:28:01.036 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: read_amp_bytes_per_bit: 0 2024-01-15T22:28:01.036 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: format_version: 2 2024-01-15T22:28:01.036 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: enable_index_compression: 1 2024-01-15T22:28:01.036 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: block_align: 0 2024-01-15T22:28:01.037 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: 2024-01-15T22:28:01.037 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-15T22:28:01.037 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-15T22:28:01.037 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.compression: NoCompression 2024-01-15T22:28:01.037 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-15T22:28:01.037 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-15T22:28:01.037 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-15T22:28:01.037 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.num_levels: 7 2024-01-15T22:28:01.037 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-15T22:28:01.037 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-15T22:28:01.037 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-15T22:28:01.037 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-15T22:28:01.037 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-15T22:28:01.037 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-15T22:28:01.037 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-15T22:28:01.038 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-15T22:28:01.038 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-15T22:28:01.038 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-15T22:28:01.038 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.compression_opts.level: 32767 2024-01-15T22:28:01.038 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-15T22:28:01.038 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-15T22:28:01.038 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-15T22:28:01.038 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.compression_opts.enabled: false 2024-01-15T22:28:01.038 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-15T22:28:01.038 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-15T22:28:01.038 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-15T22:28:01.038 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-15T22:28:01.038 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-15T22:28:01.038 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-15T22:28:01.038 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-15T22:28:01.038 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-15T22:28:01.039 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-15T22:28:01.039 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-15T22:28:01.039 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-15T22:28:01.039 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-15T22:28:01.039 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-15T22:28:01.039 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-15T22:28:01.039 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-15T22:28:01.039 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-15T22:28:01.039 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-15T22:28:01.039 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.arena_block_size: 4194304 2024-01-15T22:28:01.039 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-15T22:28:01.039 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-15T22:28:01.039 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-15T22:28:01.039 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-15T22:28:01.039 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-15T22:28:01.039 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-15T22:28:01.040 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-15T22:28:01.040 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-15T22:28:01.040 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-15T22:28:01.040 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-15T22:28:01.040 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-15T22:28:01.040 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-15T22:28:01.040 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-15T22:28:01.040 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-15T22:28:01.040 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22: 2024-01-15T22:28:01.040 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: 28:01.003+0000 7fabb8e34880 4 rocksdb: Options.table_properties_collectors: 2024-01-15T22:28:01.040 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.inplace_update_support: 0 2024-01-15T22:28:01.040 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-15T22:28:01.040 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-15T22:28:01.041 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-15T22:28:01.041 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-15T22:28:01.041 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.bloom_locality: 0 2024-01-15T22:28:01.041 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.max_successive_merges: 0 2024-01-15T22:28:01.041 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-15T22:28:01.041 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-15T22:28:01.041 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.force_consistency_checks: 0 2024-01-15T22:28:01.041 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-15T22:28:01.041 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.ttl: 2592000 2024-01-15T22:28:01.041 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.003+0000 7fabb8e34880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-15T22:28:01.041 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.004+0000 7fabb8e34880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi162/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-15T22:28:01.041 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: 2024-01-15T22:28:01.041 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.004+0000 7fabb8e34880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-01-15T22:28:01.041 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: 2024-01-15T22:28:01.041 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.004+0000 7fabb8e34880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705357681004592, "job": 1, "event": "recovery_started", "log_files": [3]} 2024-01-15T22:28:01.042 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.004+0000 7fabb8e34880 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #3 mode 2 2024-01-15T22:28:01.042 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.004+0000 7fabb8e34880 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-01-15T22:28:01.042 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.004+0000 7fabb8e34880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705357681005091, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 4, "file_size": 1437, "table_properties": {"data_size": 523, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1705357681, "oldest_key_time": 3, "file_creation_time": 0}} 2024-01-15T22:28:01.042 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.004+0000 7fabb8e34880 4 rocksdb: [version_set.cc:3826] Creating manifest 5 2024-01-15T22:28:01.042 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: 2024-01-15T22:28:01.042 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.005+0000 7fabb8e34880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705357681005504, "job": 1, "event": "recovery_finished"} 2024-01-15T22:28:01.042 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.006+0000 7fabb8e34880 4 rocksdb: DB pointer 0x558f8b5b9800 2024-01-15T22:28:01.042 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.006+0000 7fab9fc28700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-15T22:28:01.042 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.006+0000 7fab9fc28700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-15T22:28:01.042 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: ** DB Stats ** 2024-01-15T22:28:01.042 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-15T22:28:01.042 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-15T22:28:01.042 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-15T22:28:01.042 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-15T22:28:01.042 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-15T22:28:01.043 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-15T22:28:01.043 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-15T22:28:01.043 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: 2024-01-15T22:28:01.043 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: ** Compaction Stats [default] ** 2024-01-15T22:28:01.043 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-15T22:28:01.043 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-15T22:28:01.043 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.0 0.00 0.00 1 0.000 0 0 2024-01-15T22:28:01.043 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.0 0.00 0.00 1 0.000 0 0 2024-01-15T22:28:01.043 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.0 0.00 0.00 1 0.000 0 0 2024-01-15T22:28:01.043 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: 2024-01-15T22:28:01.043 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: ** Compaction Stats [default] ** 2024-01-15T22:28:01.043 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-15T22:28:01.043 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-15T22:28:01.043 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.0 0.00 0.00 1 0.000 0 0 2024-01-15T22:28:01.043 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-15T22:28:01.043 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-15T22:28:01.044 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-15T22:28:01.044 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: AddFile(Total Files): cumulative 0, interval 0 2024-01-15T22:28:01.044 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-15T22:28:01.044 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: AddFile(Keys): cumulative 0, interval 0 2024-01-15T22:28:01.044 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: Cumulative compaction: 0.00 GB write, 0.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-15T22:28:01.044 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: Interval compaction: 0.00 GB write, 0.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-15T22:28:01.044 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-15T22:28:01.044 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: 2024-01-15T22:28:01.044 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: ** File Read Latency Histogram By Level [default] ** 2024-01-15T22:28:01.044 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: 2024-01-15T22:28:01.044 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: ** Compaction Stats [default] ** 2024-01-15T22:28:01.044 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-15T22:28:01.044 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-15T22:28:01.044 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.0 0.00 0.00 1 0.000 0 0 2024-01-15T22:28:01.045 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.0 0.00 0.00 1 0.000 0 0 2024-01-15T22:28:01.045 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-15T22:28:01.045 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: 2024-01-15T22:28:01.045 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: ** Compaction Stats [default] ** 2024-01-15T22:28:01.045 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-15T22:28:01.045 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-15T22:28:01.045 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.0 0.00 0.00 1 0.000 0 0 2024-01-15T22:28:01.045 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-15T22:28:01.045 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-15T22:28:01.045 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-15T22:28:01.045 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: AddFile(Total Files): cumulative 0, interval 0 2024-01-15T22:28:01.045 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-15T22:28:01.045 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: AddFile(Keys): cumulative 0, interval 0 2024-01-15T22:28:01.046 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: Cumulative compaction: 0.00 GB write, 0.53 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-15T22:28:01.046 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-15T22:28:01.046 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-15T22:28:01.046 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: 2024-01-15T22:28:01.046 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: ** File Read Latency Histogram By Level [default] ** 2024-01-15T22:28:01.046 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: 2024-01-15T22:28:01.046 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.006+0000 7fabb8e34880 0 mon.smithi162 does not exist in monmap, will attempt to join an existing cluster 2024-01-15T22:28:01.046 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.006+0000 7fabb8e34880 0 using public_addr v2:172.21.15.162:0/0 -> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-15T22:28:01.046 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.007+0000 7fabb8e34880 0 starting mon.smithi162 rank -1 at public addrs [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] at bind addrs [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi162 fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 2024-01-15T22:28:01.046 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.008+0000 7fabb8e34880 1 mon.smithi162@-1(???) e0 preinit fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 2024-01-15T22:28:01.046 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 bash[39694]: 7ffee013c62edc65104848bb27b46d7e7eb2600d4fb52ad027bb2aa98acabffb 2024-01-15T22:28:01.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 systemd[1]: Started Ceph mon.smithi162 for eb17a8d4-b3f4-11ee-95ad-87774f69a715. 2024-01-15T22:28:01.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.037+0000 7faba242d700 0 mon.smithi162@-1(synchronizing).mds e1 new map 2024-01-15T22:28:01.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.037+0000 7faba242d700 0 mon.smithi162@-1(synchronizing).mds e1 print_map 2024-01-15T22:28:01.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: e1 2024-01-15T22:28:01.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: enable_multiple, ever_enabled_multiple: 1,1 2024-01-15T22:28:01.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-15T22:28:01.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: legacy client fscid: -1 2024-01-15T22:28:01.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: 2024-01-15T22:28:01.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: No filesystems configured 2024-01-15T22:28:01.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: 2024-01-15T22:28:01.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.037+0000 7faba242d700 1 mon.smithi162@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-01-15T22:28:01.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.037+0000 7faba242d700 1 mon.smithi162@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-01-15T22:28:01.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.038+0000 7faba242d700 1 mon.smithi162@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-01-15T22:28:01.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.038+0000 7faba242d700 1 mon.smithi162@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-01-15T22:28:01.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.038+0000 7faba242d700 1 mon.smithi162@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-01-15T22:28:01.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.038+0000 7faba242d700 1 mon.smithi162@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-01-15T22:28:01.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.038+0000 7faba242d700 1 mon.smithi162@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-01-15T22:28:01.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.039+0000 7faba242d700 0 mon.smithi162@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-01-15T22:28:01.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.039+0000 7faba242d700 0 mon.smithi162@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-01-15T22:28:01.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.039+0000 7faba242d700 0 mon.smithi162@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-01-15T22:28:01.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.039+0000 7faba242d700 0 mon.smithi162@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-01-15T22:28:01.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: cephadm 2024-01-15T22:28:01.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: 2024-01-15T22:27:57.139206+0000 mgr.smithi099.cmjeyn (mgr.14184) 2024-01-15T22:28:01.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: 9 : cephadm [INF] Deploying daemon mon.smithi162 on smithi162 2024-01-15T22:28:01.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:01 smithi162 conmon[39755]: debug 2024-01-15T22:28:01.040+0000 7faba242d700 1 mon.smithi162@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-01-15T22:28:02.660 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-15T22:28:02.660 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph mon dump -f json 2024-01-15T22:28:06.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: audit 2024-01-15T22:28:01.048994+0000 mon.smithi099 (mon.0) 168 2024-01-15T22:28:06.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-01-15T22:28:06.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: cluster 2024-01-15T22:28:01.049251+0000 mon.smithi099 (mon.0) 169 : cluster [INF] mon.smithi099 calling monitor election 2024-01-15T22:28:06.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: audit 2024-01-15 2024-01-15T22:28:06.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: T22:28:01.062865+0000 mon.smithi099 (mon.0) 170 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mon metadata", "id": "smithi162"}]: dispatch 2024-01-15T22:28:06.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: audit 2024-01- 2024-01-15T22:28:06.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: 15T22:28:02.045292+0000 mon.smithi099 (mon.0) 171 : audit [DBG] 2024-01-15T22:28:06.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mon metadata", "id": "smithi162"}]: dispatch 2024-01-15T22:28:06.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: audit 2024-01-15T22: 2024-01-15T22:28:06.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: 28:02.698457+0000 mon.smithi099 (mon.0) 172 : 2024-01-15T22:28:06.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: audit [DBG] from='mgr.? 172.21.15.162:0/4165984202' entity='mgr.smithi162.vbobpt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi162.vbobpt/crt"}]: dispatch 2024-01-15T22:28:06.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: audit 2024-01-15T22:28:03.045320+0000 2024-01-15T22:28:06.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: mon.smithi099 (mon.0) 173 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mon metadata", "id": "smithi162"}]: dispatch 2024-01-15T22:28:06.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: cluster 2024-01-15T 2024-01-15T22:28:06.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: 22:28:03.047253+0000 mon.smithi162 (mon.1) 1 2024-01-15T22:28:06.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: : cluster [INF] mon.smithi162 calling monitor election 2024-01-15T22:28:06.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: audit 2024-01-15T22:28:04. 2024-01-15T22:28:06.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: 045837+0000 mon.smithi099 (mon.0) 174 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mon metadata", "id": "smithi162"}]: dispatch 2024-01-15T22:28:06.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: 2024-01-15T22:28:06.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: audit 2024-01- 2024-01-15T22:28:06.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: 15T22:28:05.045488+0000 mon.smithi099 (mon.0) 175 : audit 2024-01-15T22:28:06.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mon metadata", "id": "smithi162"}]: dispatch 2024-01-15T22:28:06.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: cluster 2024-01-15T22:28:05.203763+0000 mgr.smithi099.cmjeyn (mgr 2024-01-15T22:28:06.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: .14184) 10 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:06.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: audit 2024-01-15T22:28: 2024-01-15T22:28:06.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: 06.045578+0000 mon.smithi099 (mon.0) 176 : audit 2024-01-15T22:28:06.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mon metadata", "id": "smithi162"}]: dispatch 2024-01-15T22:28:06.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: cluster 2024-01-15T22:28:06.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: 2024-01-15T22:28:06.076413+0000 mon.smithi099 (mon.0 2024-01-15T22:28:06.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: ) 177 : cluster [INF] mon.smithi099 is new leader, mons smithi099,smithi162 in quorum (ranks 0,1) 2024-01-15T22:28:06.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: cluster 2024-01-15T22:28: 2024-01-15T22:28:06.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: 06.079536+0000 mon.smithi099 (mon.0) 178 : cluster 2024-01-15T22:28:06.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: [DBG] monmap e2: 2 mons at {smithi099=[v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0],smithi162=[v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0]} removed_ranks: {} 2024-01-15T22:28:06.407 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: cluster 2024-01-15T22:28:06.407 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: 2024-01-15T22:28:06.091365+0000 mon.smithi099 (mon.0) 179 2024-01-15T22:28:06.407 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: : cluster [DBG] fsmap 2024-01-15T22:28:06.407 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: cluster 2024-01 2024-01-15T22:28:06.407 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: -15T22:28:06.091406+0000 mon.smithi099 (mon 2024-01-15T22:28:06.407 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: .0) 180 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-01-15T22:28:06.407 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: cluster 2024-01-15T 2024-01-15T22:28:06.407 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: 22:28:06.091633+0000 mon.smithi099 (mon.0) 2024-01-15T22:28:06.407 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: 181 : cluster [DBG] mgrmap e17: smithi099.cmjeyn(active, since 20s) 2024-01-15T22:28:06.407 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: cluster 2024-01-15T22:28: 2024-01-15T22:28:06.407 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: 06.092490+0000 mon.smithi099 (mon.0) 182 : cluster 2024-01-15T22:28:06.407 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: [DBG] Standby manager daemon smithi162.vbobpt started 2024-01-15T22:28:06.407 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: audit 2024-01-15T22:28:06.407 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: 2024-01-15T22:28:06.093559+0000 mon.smithi099 (mon 2024-01-15T22:28:06.407 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: .0) 183 : audit [DBG] from='mgr.? 172.21.15.162:0/4165984202' entity='mgr.smithi162.vbobpt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-01-15T22:28:06.407 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: cluster 2024-01-15T22 2024-01-15T22:28:06.408 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: :28:06.095063+0000 mon.smithi099 (mon.0) 184 2024-01-15T22:28:06.408 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: : cluster [INF] overall HEALTH_OK 2024-01-15T22:28:06.408 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: audit 2024-01-15T22:28:06 2024-01-15T22:28:06.408 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: .095501+0000 mon.smithi099 (mon.0) 185 : audit 2024-01-15T22:28:06.408 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: [DBG] from='mgr.? 172.21.15.162:0/4165984202' entity='mgr.smithi162.vbobpt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi162.vbobpt/key"}]: dispatch 2024-01-15T22:28:06.408 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: audit 2024-01-15T22:28:06.408 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: 2024-01-15T22:28:06.095912+0000 mon.smithi099 (mon.0) 2024-01-15T22:28:06.408 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: 186 : audit [DBG] from='mgr.? 172.21.15.162:0/4165984202' entity='mgr.smithi162.vbobpt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-01-15T22:28:06.408 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: audit 2024-01-15T22:28:06 2024-01-15T22:28:06.408 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: .097337+0000 mon.smithi099 (mon.0) 187 : audit [INF] 2024-01-15T22:28:06.408 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:06 smithi099 conmon[35292]: from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:06.422 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:06 smithi162 conmon[39755]: audit 2024-01-15T22:28:01.048994+0000 mon.smithi099 (mon.0) 168 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-01-15T22:28:06.422 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:06 smithi162 conmon[39755]: cluster 2024-01-15T22:28:01.049251+0000 mon.smithi099 (mon.0) 169 : cluster [INF] mon.smithi099 calling monitor election 2024-01-15T22:28:06.422 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:06 smithi162 conmon[39755]: audit 2024-01-15T22:28:01.062865+0000 mon.smithi099 (mon.0) 170 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mon metadata", "id": "smithi162"}]: dispatch 2024-01-15T22:28:06.422 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:06 smithi162 conmon[39755]: audit 2024-01-15T22:28:02.045292+0000 mon.smithi099 (mon.0) 171 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mon metadata", "id": "smithi162"}]: dispatch 2024-01-15T22:28:06.422 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:06 smithi162 conmon[39755]: audit 2024-01-15T22:28:02.698457+0000 mon.smithi099 (mon.0) 172 : audit [DBG] from='mgr.? 172.21.15.162:0/4165984202' entity='mgr.smithi162.vbobpt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi162.vbobpt/crt"}]: dispatch 2024-01-15T22:28:06.422 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:06 smithi162 conmon[39755]: audit 2024-01-15T22:28:03.045320+0000 mon.smithi099 (mon.0) 173 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mon metadata", "id": "smithi162"}]: dispatch 2024-01-15T22:28:06.422 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:06 smithi162 conmon[39755]: cluster 2024-01-15T22:28:03.047253+0000 mon.smithi162 (mon.1) 1 : cluster [INF] mon.smithi162 calling monitor election 2024-01-15T22:28:06.423 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:06 smithi162 conmon[39755]: audit 2024-01-15T22:28:04.045837+0000 mon.smithi099 (mon.0) 174 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mon metadata", "id": "smithi162"}]: dispatch 2024-01-15T22:28:06.423 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:06 smithi162 conmon[39755]: 2024-01-15T22:28:06.423 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:06 smithi162 conmon[39755]: audit 2024-01-15T22:28:05.045488+0000 mon.smithi099 (mon.0) 175 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mon metadata", "id": "smithi162"}]: dispatch 2024-01-15T22:28:06.423 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:06 smithi162 conmon[39755]: cluster 2024-01-15T22:28:05.203763+0000 mgr.smithi099.cmjeyn (mgr.14184) 10 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:06.423 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:06 smithi162 conmon[39755]: audit 2024-01-15T22:28:06.045578+0000 mon.smithi099 (mon.0) 176 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mon metadata", "id": "smithi162"}]: dispatch 2024-01-15T22:28:06.423 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:06 smithi162 conmon[39755]: cluster 2024-01-15T22:28:06.076413+0000 mon.smithi099 (mon.0) 177 : cluster [INF] mon.smithi099 is new leader, mons smithi099,smithi162 in quorum (ranks 0,1) 2024-01-15T22:28:06.423 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:06 smithi162 conmon[39755]: cluster 2024-01-15T22:28:06.079536+0000 mon.smithi099 (mon.0) 178 : cluster [DBG] monmap e2: 2 mons at {smithi099=[v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0],smithi162=[v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0]} removed_ranks: {} 2024-01-15T22:28:06.423 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:06 smithi162 conmon[39755]: cluster 2024-01-15T22:28:06.091365+0000 mon.smithi099 (mon.0) 179 : cluster [DBG] fsmap 2024-01-15T22:28:06.423 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:06 smithi162 conmon[39755]: cluster 2024-01-15T22:28:06.091406+0000 mon.smithi099 (mon.0) 180 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-01-15T22:28:06.423 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:06 smithi162 conmon[39755]: cluster 2024-01-15T22:28:06.091633+0000 mon.smithi099 (mon.0) 181 : cluster [DBG] mgrmap e17: smithi099.cmjeyn(active, since 20s) 2024-01-15T22:28:06.423 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:06 smithi162 conmon[39755]: cluster 2024-01-15T22:28:06.092490+0000 mon.smithi099 (mon.0) 182 : cluster [DBG] Standby manager daemon smithi162.vbobpt started 2024-01-15T22:28:06.423 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:06 smithi162 conmon[39755]: audit 2024-01-15T22:28:06.093559+0000 mon.smithi099 (mon.0) 183 : audit [DBG] from='mgr.? 172.21.15.162:0/4165984202' entity='mgr.smithi162.vbobpt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-01-15T22:28:06.423 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:06 smithi162 conmon[39755]: cluster 2024-01-15T22:28:06.095063+0000 mon.smithi099 (mon.0) 184 : cluster [INF] overall HEALTH_OK 2024-01-15T22:28:06.423 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:06 smithi162 conmon[39755]: audit 2024-01-15T22:28:06.095501+0000 mon.smithi099 (mon.0) 185 : audit [DBG] from='mgr.? 172.21.15.162:0/4165984202' entity='mgr.smithi162.vbobpt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi162.vbobpt/key"}]: dispatch 2024-01-15T22:28:06.423 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:06 smithi162 conmon[39755]: audit 2024-01-15T22:28:06.095912+0000 mon.smithi099 (mon.0) 186 : audit [DBG] from='mgr.? 172.21.15.162:0/4165984202' entity='mgr.smithi162.vbobpt' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-01-15T22:28:06.423 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:06 smithi162 conmon[39755]: audit 2024-01-15T22:28:06.097337+0000 mon.smithi099 (mon.0) 187 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:07.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:07 smithi099 conmon[35292]: cephadm 2024-01-15T22:28:06.099103+0000 mgr.smithi099.cmjeyn (mgr.14184) 11 : cephadm 2024-01-15T22:28:07.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:07 smithi099 conmon[35292]: [INF] Deploying daemon node-exporter.smithi162 on smithi162 2024-01-15T22:28:07.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:07 smithi099 conmon[35292]: cluster 2024-01-15T22:28:06.152139+0000 mon.smithi099 (mon.0) 188 : cluster [DBG] mgrmap e18: smithi099.cmjeyn(active, since 20s), standbys: smithi162.vbobpt 2024-01-15T22:28:07.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:07 smithi099 conmon[35292]: audit 2024-01-15T22:28:06.152353+0000 mon.smithi099 (mon.0) 189 : audit 2024-01-15T22:28:07.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:07 smithi099 conmon[35292]: [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mgr metadata", "who": "smithi162.vbobpt", "id": "smithi162.vbobpt"}]: dispatch 2024-01-15T22:28:07.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:07 smithi099 conmon[35292]: audit 2024-01-15T22:28:07.045993+0000 mon.smithi099 (mon.0) 190 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mon metadata", "id": "smithi162"}]: dispatch 2024-01-15T22:28:07.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:07 smithi099 conmon[35292]: 2024-01-15T22:28:07.406 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-01-15T22:28:07.406 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":2,"fsid":"eb17a8d4-b3f4-11ee-95ad-87774f69a715","modified":"2024-01-15T22:28:01.046354Z","created":"2024-01-15T22:25:52.892888Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi099","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:3300","nonce":0},{"type":"v1","addr":"172.21.15.99:6789","nonce":0}]},"addr":"172.21.15.99:6789/0","public_addr":"172.21.15.99:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi162","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:3300","nonce":0},{"type":"v1","addr":"172.21.15.162:6789","nonce":0}]},"addr":"172.21.15.162:6789/0","public_addr":"172.21.15.162:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-01-15T22:28:07.408 INFO:teuthology.orchestra.run.smithi162.stderr:dumped monmap epoch 2 2024-01-15T22:28:07.432 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:07 smithi162 conmon[39755]: cephadm 2024-01-15T22:28:06.099103+0000 mgr.smithi099.cmjeyn (mgr.14184) 11 : cephadm [INF] Deploying daemon node-exporter.smithi162 on smithi162 2024-01-15T22:28:07.432 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:07 smithi162 conmon[39755]: cluster 2024-01- 2024-01-15T22:28:07.432 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:07 smithi162 conmon[39755]: 15T22:28:06.152139+0000 mon.smithi099 (mon.0) 188 : cluster [DBG] mgrmap e18: smithi099.cmjeyn(active, since 20s), standbys: smithi162.vbobpt 2024-01-15T22:28:07.432 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:07 smithi162 conmon[39755]: audit 2024-01-15T22:28:06.152353+0000 mon.smithi099 (mon.0) 189 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mgr metadata", "who": "smithi162.vbobpt", "id": "smithi162.vbobpt"}]: dispatch 2024-01-15T22:28:07.432 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:07 smithi162 conmon[39755]: audit 2024-01-15T22:28:07.045993+0000 mon.smithi099 (mon.0) 190 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mon metadata", "id": "smithi162"}]: dispatch 2024-01-15T22:28:08.010 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-01-15T22:28:08.010 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph config generate-minimal-conf 2024-01-15T22:28:08.279 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:08 smithi162 conmon[39755]: cluster 2024-01-15T22:28:07 2024-01-15T22:28:08.279 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:08 smithi162 conmon[39755]: .204190+0000 mgr.smithi099.cmjeyn (mgr.14184) 12 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:08.279 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:08 smithi162 conmon[39755]: audit 2024-01-15T22:28:07.405146+0000 mon.smithi099 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.162:0/1106644554' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-15T22:28:08.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:08 smithi099 conmon[35292]: cluster 2024-01-15T22:28:07.204190+0000 mgr.smithi099.cmjeyn (mgr.14184) 12 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:08.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:08 smithi099 conmon[35292]: audit 2024-01-15T22:28:07.405146+0000 mon.smithi099 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.162:0/1106644554' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-15T22:28:09.865 INFO:teuthology.orchestra.run.smithi099.stdout:# minimal ceph.conf for eb17a8d4-b3f4-11ee-95ad-87774f69a715 2024-01-15T22:28:09.865 INFO:teuthology.orchestra.run.smithi099.stdout:[global] 2024-01-15T22:28:09.865 INFO:teuthology.orchestra.run.smithi099.stdout: fsid = eb17a8d4-b3f4-11ee-95ad-87774f69a715 2024-01-15T22:28:09.866 INFO:teuthology.orchestra.run.smithi099.stdout: mon_host = [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-01-15T22:28:10.425 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-01-15T22:28:10.426 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-01-15T22:28:10.426 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/ceph/ceph.conf 2024-01-15T22:28:10.456 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-01-15T22:28:10.456 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-15T22:28:10.520 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:10 smithi099 conmon[35292]: cluster 2024-01-15T22:28:09.204581+0000 mgr.smithi099.cmjeyn (mgr.14184) 13 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:10.520 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:10 smithi099 conmon[35292]: audit 2024-01-15T22:28:09.865857+0000 mon.smithi099 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.99:0/2758215003' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:28:10.523 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-01-15T22:28:10.523 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/ceph/ceph.conf 2024-01-15T22:28:10.558 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:10 smithi162 conmon[39755]: cluster 2024-01-15T22:28:09.204581+0000 mgr.smithi099.cmjeyn (mgr.14184) 13 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:10.558 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:10 smithi162 conmon[39755]: audit 2024-01-15T22:28:09.865857+0000 mon.smithi099 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.99:0/2758215003' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:28:10.563 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-01-15T22:28:10.563 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-15T22:28:10.632 INFO:tasks.cephadm:Deploying OSDs... 2024-01-15T22:28:10.632 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-01-15T22:28:10.633 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/scratch_devs of=/dev/stdout 2024-01-15T22:28:10.649 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-01-15T22:28:10.649 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/nvme1n1 2024-01-15T22:28:10.707 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/nvme1n1 2024-01-15T22:28:10.707 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-15T22:28:10.707 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 200143 Links: 1 Device type: 103,1 2024-01-15T22:28:10.707 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-15T22:28:10.707 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:nvme_device_t:s0 2024-01-15T22:28:10.707 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-01-15 22:26:54.440971004 +0000 2024-01-15T22:28:10.707 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-01-15 22:24:56.906587264 +0000 2024-01-15T22:28:10.707 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-01-15 22:24:56.906587264 +0000 2024-01-15T22:28:10.707 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2024-01-15T22:28:10.707 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-01-15T22:28:10.772 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-01-15T22:28:10.772 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-01-15T22:28:10.772 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000283392 s, 1.8 MB/s 2024-01-15T22:28:10.773 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-01-15T22:28:10.831 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/nvme2n1 2024-01-15T22:28:10.888 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/nvme2n1 2024-01-15T22:28:10.889 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-15T22:28:10.889 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 204055 Links: 1 Device type: 103,2 2024-01-15T22:28:10.889 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-15T22:28:10.889 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:nvme_device_t:s0 2024-01-15T22:28:10.889 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-01-15 22:26:54.450970781 +0000 2024-01-15T22:28:10.889 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-01-15 22:24:57.058583878 +0000 2024-01-15T22:28:10.889 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-01-15 22:24:57.058583878 +0000 2024-01-15T22:28:10.889 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2024-01-15T22:28:10.889 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-01-15T22:28:10.954 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-01-15T22:28:10.954 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-01-15T22:28:10.954 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000266035 s, 1.9 MB/s 2024-01-15T22:28:10.955 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-01-15T22:28:11.013 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/nvme3n1 2024-01-15T22:28:11.071 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/nvme3n1 2024-01-15T22:28:11.071 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-15T22:28:11.071 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 205051 Links: 1 Device type: 103,3 2024-01-15T22:28:11.071 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-15T22:28:11.071 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:nvme_device_t:s0 2024-01-15T22:28:11.071 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-01-15 22:26:54.460970559 +0000 2024-01-15T22:28:11.071 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-01-15 22:24:57.208580537 +0000 2024-01-15T22:28:11.071 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-01-15 22:24:57.208580537 +0000 2024-01-15T22:28:11.071 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2024-01-15T22:28:11.072 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-01-15T22:28:11.137 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-01-15T22:28:11.137 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-01-15T22:28:11.137 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000217222 s, 2.4 MB/s 2024-01-15T22:28:11.138 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-01-15T22:28:11.196 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/nvme4n1 2024-01-15T22:28:11.254 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/nvme4n1 2024-01-15T22:28:11.254 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-15T22:28:11.254 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 204182 Links: 1 Device type: 103,4 2024-01-15T22:28:11.254 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-15T22:28:11.254 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:nvme_device_t:s0 2024-01-15T22:28:11.254 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-01-15 22:26:54.471970314 +0000 2024-01-15T22:28:11.254 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-01-15 22:24:57.357577217 +0000 2024-01-15T22:28:11.254 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-01-15 22:24:57.357577217 +0000 2024-01-15T22:28:11.254 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2024-01-15T22:28:11.254 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-01-15T22:28:11.318 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-01-15T22:28:11.318 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-01-15T22:28:11.318 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000233622 s, 2.2 MB/s 2024-01-15T22:28:11.320 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-01-15T22:28:11.377 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-01-15T22:28:11.377 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/scratch_devs of=/dev/stdout 2024-01-15T22:28:11.394 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-01-15T22:28:11.395 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/nvme1n1 2024-01-15T22:28:11.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:11 smithi099 conmon[35292]: audit 2024-01-15T22:28:10.355688+0000 mon.smithi099 (mon.0) 193 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:11.452 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/nvme1n1 2024-01-15T22:28:11.453 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-15T22:28:11.453 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 6h/6d Inode: 201012 Links: 1 Device type: 103,1 2024-01-15T22:28:11.453 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-15T22:28:11.453 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:nvme_device_t:s0 2024-01-15T22:28:11.453 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-01-15 22:27:48.866263573 +0000 2024-01-15T22:28:11.453 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-01-15 22:24:58.566615278 +0000 2024-01-15T22:28:11.453 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-01-15 22:24:58.566615278 +0000 2024-01-15T22:28:11.453 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: - 2024-01-15T22:28:11.453 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-01-15T22:28:11.519 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-01-15T22:28:11.520 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-01-15T22:28:11.520 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000235641 s, 2.2 MB/s 2024-01-15T22:28:11.521 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-01-15T22:28:11.580 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/nvme2n1 2024-01-15T22:28:11.626 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:11 smithi162 conmon[39755]: audit 2024-01-15T22:28:10.355688+0000 mon.smithi099 (mon. 2024-01-15T22:28:11.626 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:11 smithi162 conmon[39755]: 0) 193 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:11.638 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/nvme2n1 2024-01-15T22:28:11.638 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-15T22:28:11.638 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 6h/6d Inode: 200644 Links: 1 Device type: 103,2 2024-01-15T22:28:11.638 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-15T22:28:11.638 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:nvme_device_t:s0 2024-01-15T22:28:11.638 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-01-15 22:27:48.877263357 +0000 2024-01-15T22:28:11.638 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-01-15 22:24:58.717612309 +0000 2024-01-15T22:28:11.638 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-01-15 22:24:58.717612309 +0000 2024-01-15T22:28:11.638 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: - 2024-01-15T22:28:11.638 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-01-15T22:28:11.703 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-01-15T22:28:11.704 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-01-15T22:28:11.704 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000237708 s, 2.2 MB/s 2024-01-15T22:28:11.705 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-01-15T22:28:11.763 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/nvme3n1 2024-01-15T22:28:11.821 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/nvme3n1 2024-01-15T22:28:11.821 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-15T22:28:11.821 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 6h/6d Inode: 206945 Links: 1 Device type: 103,3 2024-01-15T22:28:11.821 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-15T22:28:11.821 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:nvme_device_t:s0 2024-01-15T22:28:11.821 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-01-15 22:27:48.887263160 +0000 2024-01-15T22:28:11.822 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-01-15 22:24:58.868609341 +0000 2024-01-15T22:28:11.822 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-01-15 22:24:58.868609341 +0000 2024-01-15T22:28:11.822 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: - 2024-01-15T22:28:11.822 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-01-15T22:28:11.888 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-01-15T22:28:11.888 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-01-15T22:28:11.888 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000216758 s, 2.4 MB/s 2024-01-15T22:28:11.890 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-01-15T22:28:11.947 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/nvme4n1 2024-01-15T22:28:12.005 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/nvme4n1 2024-01-15T22:28:12.005 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-15T22:28:12.005 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 6h/6d Inode: 207001 Links: 1 Device type: 103,4 2024-01-15T22:28:12.005 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-15T22:28:12.005 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:nvme_device_t:s0 2024-01-15T22:28:12.005 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-01-15 22:27:48.898262944 +0000 2024-01-15T22:28:12.005 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-01-15 22:24:59.019606372 +0000 2024-01-15T22:28:12.005 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-01-15 22:24:59.019606372 +0000 2024-01-15T22:28:12.005 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: - 2024-01-15T22:28:12.006 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-01-15T22:28:12.074 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-01-15T22:28:12.074 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-01-15T22:28:12.074 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000250167 s, 2.0 MB/s 2024-01-15T22:28:12.076 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-01-15T22:28:12.134 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph orch apply osd --all-available-devices 2024-01-15T22:28:12.618 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:12 smithi162 conmon[39755]: cluster 2024-01-15T22:28:11.204981+0000 mgr.smithi099.cmjeyn (mgr.14184) 14 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:12.618 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:12 smithi162 conmon[39755]: audit 2024-01-15T22:28:11.814288+0000 mon.smithi099 (mon.0) 194 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:12.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:12 smithi099 conmon[35292]: cluster 2024-01-15T22:28:11.204981+0000 mgr.smithi099.cmjeyn (mgr.14184) 14 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:12.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:12 smithi099 conmon[35292]: audit 2024-01-15T22:28:11.814288+0000 mon.smithi099 (mon.0) 194 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:13.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:13 smithi099 conmon[35292]: cephadm 2024-01-15T22:28:11.827509+0000 mgr.smithi099.cmjeyn (mgr.14184) 15 : cephadm [INF] Deploying daemon prometheus.smithi162 on smithi162 2024-01-15T22:28:13.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:13 smithi162 conmon[39755]: cephadm 2024-01-15T22:28:11.827509+0000 mgr.smithi099.cmjeyn (mgr.14184) 15 : cephadm [INF] Deploying daemon prometheus.smithi162 on smithi162 2024-01-15T22:28:14.119 INFO:teuthology.orchestra.run.smithi162.stdout:Scheduled osd.all-available-devices update... 2024-01-15T22:28:14.446 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:14 smithi162 conmon[39755]: cluster 2024-01-15T22:28:13.205275+0000 mgr.smithi099.cmjeyn (mgr.14184) 16 : cluster 2024-01-15T22:28:14.446 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:14 smithi162 conmon[39755]: [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:14.446 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:14 smithi162 conmon[39755]: audit 2024-01-15T22:28:14.119915+0000 mon.smithi099 (mon.0) 195 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:14.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:14 smithi099 conmon[35292]: cluster 2024-01-15T22:28:13.205275+0000 mgr.smithi099.cmjeyn (mgr.14184) 16 : cluster 2024-01-15T22:28:14.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:14 smithi099 conmon[35292]: [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:14.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:14 smithi099 conmon[35292]: audit 2024-01-15T22:28:14.119915+0000 mon.smithi099 (mon.0) 195 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:14.677 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-01-15T22:28:14.677 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:28:15.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:15 smithi099 conmon[35292]: audit 2024-01-15T22:28:14.114724+0000 mgr.smithi099.cmjeyn 2024-01-15T22:28:15.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:15 smithi099 conmon[35292]: (mgr.14184) 17 : audit [DBG] from='client.24101 -' entity='client.admin' cmd=[{"prefix": "orch apply osd", "all_available_devices": true, "target": ["mon-mgr", ""]}]: dispatch 2024-01-15T22:28:15.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:15 smithi099 conmon[35292]: cephadm 2024-01-15T22:28:14.116333+0000 mgr.smithi099.cmjeyn (mgr.14184) 18 : cephadm [INF] Marking host: smithi099 for OSDSpec preview refresh. 2024-01-15T22:28:15.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:15 smithi099 conmon[35292]: cephadm 2024-01-15T22:28:14.116405+0000 mgr.smithi099.cmjeyn (mgr.14184) 19 : cephadm [INF] Marking host: smithi162 for OSDSpec preview refresh. 2024-01-15T22:28:15.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:15 smithi099 conmon[35292]: cephadm 2024-01-15T22:28:14.116757+0000 mgr.smithi099.cmjeyn (mgr.14184) 20 : cephadm [INF] Saving service osd.all-available-devices spec with placement * 2024-01-15T22:28:15.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:15 smithi099 conmon[35292]: audit 2024-01-15T22:28:15.359877+0000 mon.smithi099 (mon.0) 196 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:15.696 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:15 smithi162 conmon[39755]: audit 2024-01-15T22:28:14.114724+0000 mgr.smithi099.cmjeyn (mgr.14184) 17 : audit [DBG] from='client.24101 -' entity='client.admin' cmd=[{"prefix": "orch apply osd", "all_available_devices": true, "target": ["mon-mgr", ""]}]: dispatch 2024-01-15T22:28:15.696 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:15 smithi162 conmon[39755]: 2024-01-15T22:28:15.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:15 smithi162 conmon[39755]: cephadm 2024-01-15T22:28:14.116333+0000 mgr.smithi099.cmjeyn (mgr.14184) 18 : cephadm [INF] Marking host: smithi099 for OSDSpec preview refresh. 2024-01-15T22:28:15.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:15 smithi162 conmon[39755]: cephadm 2024-01-15T22:28:14.116405 2024-01-15T22:28:15.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:15 smithi162 conmon[39755]: +0000 mgr.smithi099.cmjeyn (mgr.14184) 19 : cephadm [INF] Marking host: smithi162 for OSDSpec preview refresh. 2024-01-15T22:28:15.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:15 smithi162 conmon[39755]: cephadm 2024-01- 2024-01-15T22:28:15.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:15 smithi162 conmon[39755]: 15T22:28:14.116757+0000 mgr.smithi099.cmjeyn (mgr.14184) 20 2024-01-15T22:28:15.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:15 smithi162 conmon[39755]: : cephadm [INF] Saving service osd.all-available-devices spec with placement * 2024-01-15T22:28:15.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:15 smithi162 conmon[39755]: audit 2024-01-15T22:28:15.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:15 smithi162 conmon[39755]: 2024-01-15T22:28:15.359877 2024-01-15T22:28:15.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:15 smithi162 conmon[39755]: +0000 mon.smithi099 (mon.0) 196 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:16.520 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:28:16.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:16 smithi099 conmon[35292]: cluster 2024-01-15T22:28:15.205569+0000 mgr.smithi099.cmjeyn (mgr.14184) 21 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:16.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:16 smithi099 conmon[35292]: 2024-01-15T22:28:16.696 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:16 smithi162 conmon[39755]: cluster 2024-01-15T22:28:15.205569+0000 mgr.smithi099.cmjeyn (mgr.14184) 21 : cluster [DBG] 2024-01-15T22:28:16.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:16 smithi162 conmon[39755]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:17.050 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2024-01-15T22:28:17.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:17 smithi099 conmon[35292]: audit 2024-01-15T22:28:16.520710+0000 mon.smithi099 (mon.0) 197 : audit [DBG] from='client.? 172.21.15.99:0/407132423' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:28:17.696 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:17 smithi162 conmon[39755]: audit 2024-01-15T22:28:16.520710+0000 mon.smithi099 (mon.0) 197 : audit [DBG] from='client.? 172.21.15.99:0/407132423' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:28:18.051 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:28:18.642 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:18 smithi099 conmon[35292]: cluster 2024-01-15T22:28:17.205880+0000 mgr.smithi099.cmjeyn (mgr 2024-01-15T22:28:18.642 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:18 smithi099 conmon[35292]: .14184) 22 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:18.696 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:18 smithi162 conmon[39755]: cluster 2024-01-15T22:28:17.205880+0000 mgr.smithi099.cmjeyn (mgr.14184) 22 : cluster 2024-01-15T22:28:18.696 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:18 smithi162 conmon[39755]: [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:19.800 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:28:20.374 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2024-01-15T22:28:20.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:20 smithi099 conmon[35292]: cluster 2024-01-15T22:28:19.206091+0000 mgr.smithi099.cmjeyn (mgr.14184) 23 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:20.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:20 smithi099 conmon[35292]: audit 2024-01-15T22:28:19.800730+0000 mon.smithi099 (mon.0) 198 : audit [DBG] from='client.? 172.21.15.99:0/1102552554' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:28:20.696 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:20 smithi162 conmon[39755]: cluster 2024-01-15T22:28:19.206091+0000 mgr.smithi099.cmjeyn (mgr.14184) 23 : cluster 2024-01-15T22:28:20.696 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:20 smithi162 conmon[39755]: [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:20.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:20 smithi162 conmon[39755]: audit 2024-01-15T22:28:19.800730+0000 mon.smithi099 (mon.0) 198 : audit [DBG] from='client.? 172.21.15.99:0/1102552554' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:28:21.375 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:28:22.575 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:22 smithi099 conmon[35292]: cluster 2024-01-15T22:28:21.206462+0000 mgr.smithi099.cmjeyn (mgr.14184) 24 : 2024-01-15T22:28:22.575 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:22 smithi099 conmon[35292]: cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:22.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:22 smithi162 conmon[39755]: cluster 2024-01-15T22:28:21.206462+0000 mgr.smithi099.cmjeyn (mgr.14184) 24 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:23.155 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:28:23.600 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:23 smithi162 conmon[39755]: audit 2024-01-15T22:28:23.155794+0000 mon.smithi099 (mon.0) 2024-01-15T22:28:23.600 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:23 smithi162 conmon[39755]: 199 : audit [DBG] from='client.? 172.21.15.99:0/502630548' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:28:23.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:23 smithi099 conmon[35292]: audit 2024-01-15T22:28:23.155794+0000 mon.smithi099 (mon.0) 199 : audit [DBG] 2024-01-15T22:28:23.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:23 smithi099 conmon[35292]: from='client.? 172.21.15.99:0/502630548' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:28:23.682 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2024-01-15T22:28:24.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:24 smithi099 conmon[35292]: cluster 2024-01-15T22:28:23.206761+0000 mgr.smithi099.cmjeyn (mgr.14184) 25 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:24.683 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:28:24.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:24 smithi162 conmon[39755]: cluster 2024-01-15T22:28:23.206761+0000 mgr.smithi099.cmjeyn (mgr.14184) 25 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:26.138 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:25 smithi099 conmon[35292]: audit 2024-01-15T22:28:24.880052+0000 mon.smithi099 (mon 2024-01-15T22:28:26.138 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:25 smithi099 conmon[35292]: .0) 200 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:26.138 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:25 smithi099 conmon[35292]: audit 2024-01-15T22:28:24.881299+0000 mon.smithi099 (mon.0) 201 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-15T22:28:26.138 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:25 smithi099 conmon[35292]: audit 2024-01-15T22:28:24.887123+0000 mon.smithi099 (mon.0) 202 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:26.138 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:25 smithi099 conmon[35292]: cluster 2024-01-15T22:28:25.206980+0000 mgr.smithi099.cmjeyn (mgr.14184) 26 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:26.138 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:25 smithi099 conmon[35292]: audit 2024-01-15T22:28:25.374856+0000 mon.smithi099 (mon.0) 203 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:26.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:25 smithi162 conmon[39755]: audit 2024-01-15T22:28:24.880052+0000 mon.smithi099 (mon.0) 200 2024-01-15T22:28:26.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:25 smithi162 conmon[39755]: : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:26.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:25 smithi162 conmon[39755]: audit 2024-01-15T22:28:24.881299+0000 mon.smithi099 (mon.0) 201 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-15T22:28:26.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:25 smithi162 conmon[39755]: audit 2024-01-15T22:28:24.887123+0000 mon.smithi099 (mon.0) 202 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:26.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:25 smithi162 conmon[39755]: cluster 2024-01-15T22:28:25.206980+0000 mgr.smithi099.cmjeyn (mgr.14184) 26 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:26.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:25 smithi162 conmon[39755]: audit 2024-01-15T22:28:25.374856+0000 mon.smithi099 (mon.0) 203 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:26.440 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:28:27.023 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2024-01-15T22:28:27.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:26 smithi099 conmon[35292]: audit 2024-01-15T22:28:26.440894+0000 mon.smithi099 (mon.0 2024-01-15T22:28:27.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:26 smithi099 conmon[35292]: ) 204 : audit [DBG] from='client.? 172.21.15.99:0/2894961048' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:28:27.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:26 smithi099 conmon[35292]: audit 2024-01-15T22:28:26.698039+0000 mon.smithi099 (mon.0) 205 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:27.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:26 smithi099 conmon[35292]: audit 2024-01-15T22:28:26.704304+0000 mon.smithi099 (mon.0) 206 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:27.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:26 smithi099 conmon[35292]: audit 2024-01-15T22:28:26.704942+0000 mon.smithi099 (mon.0) 207 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:28:27.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:26 smithi099 conmon[35292]: audit 2024-01-15T22:28:26.705476+0000 mon.smithi099 (mon.0) 208 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-15T22:28:27.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:26 smithi099 conmon[35292]: cephadm 2024-01-15T22:28:26.706322+0000 mgr.smithi099.cmjeyn 2024-01-15T22:28:27.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:26 smithi099 conmon[35292]: (mgr.14184) 27 : cephadm [INF] Updating smithi099:/etc/ceph/ceph.conf 2024-01-15T22:28:27.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:26 smithi099 conmon[35292]: cephadm 2024-01-15T22:28:26.706532+0000 mgr.smithi099.cmjeyn (mgr.14184) 2024-01-15T22:28:27.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:26 smithi099 conmon[35292]: 28 : cephadm [INF] Updating smithi162:/etc/ceph/ceph.conf 2024-01-15T22:28:27.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:26 smithi099 conmon[35292]: audit 2024-01-15T22:28 2024-01-15T22:28:27.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:26 smithi099 conmon[35292]: :26.732555+0000 mon.smithi099 (mon.0) 209 : audit 2024-01-15T22:28:27.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:26 smithi099 conmon[35292]: [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:27.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:26 smithi099 conmon[35292]: audit 2024 2024-01-15T22:28:27.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:26 smithi099 conmon[35292]: -01-15T22:28:26.879464+0000 mon.smithi099 2024-01-15T22:28:27.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:26 smithi099 conmon[35292]: (mon.0) 210 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:27.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:26 smithi099 conmon[35292]: audit 2024-01-15T22:28:27.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:26 smithi099 conmon[35292]: 2024-01-15T22:28:26 2024-01-15T22:28:27.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:26 smithi099 conmon[35292]: .882452+0000 mon.smithi099 ( 2024-01-15T22:28:27.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:26 smithi099 conmon[35292]: mon.0) 211 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:27.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:26 smithi099 conmon[35292]: audit 2024-01-15T22 2024-01-15T22:28:27.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:26 smithi099 conmon[35292]: :28:26.885738+0000 mon.smithi099 (mon.0) 212 2024-01-15T22:28:27.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:26 smithi099 conmon[35292]: : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-15T22:28:27.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:26 smithi099 conmon[35292]: audit 2024-01-15T22:28 2024-01-15T22:28:27.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:26 smithi099 conmon[35292]: :26.888470+0000 mon.smithi099 (mon.0) 213 : 2024-01-15T22:28:27.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:26 smithi099 conmon[35292]: audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-15T22:28:27.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:26 smithi099 conmon[35292]: audit 2024-01-15T22:28: 2024-01-15T22:28:27.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:26 smithi099 conmon[35292]: 26.889339+0000 mon.smithi099 (mon.0) 214 : audit 2024-01-15T22:28:27.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:26 smithi099 conmon[35292]: [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-15T22:28:27.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:26 smithi099 conmon[35292]: audit 2024-01-15T22:28:26. 2024-01-15T22:28:27.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:26 smithi099 conmon[35292]: 889533+0000 mon.smithi099 (mon.0) 215 : audit [DBG] 2024-01-15T22:28:27.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:26 smithi099 conmon[35292]: from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:28:27.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:26 smithi099 conmon[35292]: audit 2024-01-15T22:28:26.890002+0000 2024-01-15T22:28:27.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:26 smithi099 conmon[35292]: mon.smithi099 (mon.0) 216 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:28:27.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:26 smithi099 conmon[35292]: 2024-01-15T22:28:27.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:26 smithi162 conmon[39755]: audit 2024-01-15T22:28:26.440894+0000 mon.smithi099 (mon.0) 204 2024-01-15T22:28:27.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:26 smithi162 conmon[39755]: : audit [DBG] from='client.? 172.21.15.99:0/2894961048' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:28:27.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:26 smithi162 conmon[39755]: audit 2024-01-15T22:28:26.698039+0000 mon.smithi099 (mon.0) 205 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:27.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:26 smithi162 conmon[39755]: audit 2024-01-15T22:28:26.704304+0000 mon.smithi099 (mon.0) 206 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:27.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:26 smithi162 conmon[39755]: audit 2024-01-15T 2024-01-15T22:28:27.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:26 smithi162 conmon[39755]: 22:28:26.704942+0000 mon.smithi099 (mon.0) 207 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:28:27.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:26 smithi162 conmon[39755]: audit 2024-01-15T22: 2024-01-15T22:28:27.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:26 smithi162 conmon[39755]: 28:26.705476+0000 mon.smithi099 (mon.0) 208 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-15T22:28:27.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:26 smithi162 conmon[39755]: cephadm 2024-01- 2024-01-15T22:28:27.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:26 smithi162 conmon[39755]: 15T22:28:26.706322+0000 mgr.smithi099.cmjeyn (mgr.14184) 27 2024-01-15T22:28:27.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:26 smithi162 conmon[39755]: : cephadm [INF] Updating smithi099:/etc/ceph/ceph.conf 2024-01-15T22:28:27.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:26 smithi162 conmon[39755]: cephadm 2024-01-15T22:28:27.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:26 smithi162 conmon[39755]: 2024-01-15T22:28:26.706532+0000 mgr.smithi099.cmjeyn (mgr.14184) 28 : 2024-01-15T22:28:27.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:26 smithi162 conmon[39755]: cephadm [INF] Updating smithi162:/etc/ceph/ceph.conf 2024-01-15T22:28:27.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:26 smithi162 conmon[39755]: audit 2024-01-15T22:28:27.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:26 smithi162 conmon[39755]: 2024-01-15T22:28:26.732555+0000 mon.smithi099 (mon.0 2024-01-15T22:28:27.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:26 smithi162 conmon[39755]: ) 209 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:27.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:26 smithi162 conmon[39755]: audit 2024-01-15T22:28:27.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:26 smithi162 conmon[39755]: 2024-01-15T22:28:26.879464+0000 mon.smithi099 (mon 2024-01-15T22:28:27.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:26 smithi162 conmon[39755]: .0) 210 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:27.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:26 smithi162 conmon[39755]: audit 2024-01-15T22:28:26 2024-01-15T22:28:27.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:26 smithi162 conmon[39755]: .882452+0000 mon.smithi099 (mon.0) 211 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:27.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:26 smithi162 conmon[39755]: audit 2024-01 2024-01-15T22:28:27.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:26 smithi162 conmon[39755]: -15T22:28:26.885738+0000 mon.smithi099 ( 2024-01-15T22:28:27.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:26 smithi162 conmon[39755]: mon.0) 212 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-15T22:28:27.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:26 smithi162 conmon[39755]: audit 2024-01-15T22:28 2024-01-15T22:28:27.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:26 smithi162 conmon[39755]: :26.888470+0000 mon.smithi099 (mon.0) 213 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-15T22:28:27.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:26 smithi162 conmon[39755]: 2024-01-15T22:28:27.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:26 smithi162 conmon[39755]: audit 2024- 2024-01-15T22:28:27.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:26 smithi162 conmon[39755]: 01-15T22:28:26.889339+0000 mon.smithi099 (mon.0) 214 : audit 2024-01-15T22:28:27.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:26 smithi162 conmon[39755]: [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-15T22:28:27.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:26 smithi162 conmon[39755]: audit 2024 2024-01-15T22:28:27.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:26 smithi162 conmon[39755]: -01-15T22:28:26.889533+0000 mon.smithi099 (mon.0) 215 : audit [DBG] 2024-01-15T22:28:27.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:26 smithi162 conmon[39755]: from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:28:27.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:26 smithi162 conmon[39755]: audit 2024-01 2024-01-15T22:28:27.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:26 smithi162 conmon[39755]: -15T22:28:26.890002+0000 mon.smithi099 (mon.0) 216 : audit [DBG] 2024-01-15T22:28:27.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:26 smithi162 conmon[39755]: from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:28:28.024 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:28:28.051 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:27 smithi099 conmon[35292]: cluster 2024-01-15T22:28:27.207194+0000 mgr.smithi099.cmjeyn (mgr.14184) 29 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:28.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:27 smithi162 conmon[39755]: cluster 2024-01-15T22:28:27.207194+0000 mgr.smithi099.cmjeyn (mgr.14184) 29 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:29.906 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:28:30.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:30 smithi099 conmon[35292]: cluster 2024-01-15T22:28:29.207409+0000 mgr.smithi099.cmjeyn (mgr.14184) 30 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:30.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:30 smithi099 conmon[35292]: audit 2024-01-15T22:28:29.497000+0000 mon.smithi099 (mon.0) 217 : audit [INF] from='client.? 172.21.15.99:0/2519223484' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b377d95f-37bd-44b3-a231-13d69245430a"}]: dispatch 2024-01-15T22:28:30.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:30 smithi099 conmon[35292]: audit 2024-01-15T22:28:29.502453+0000 mon.smithi099 (mon.0) 218 : audit [INF] from='client.? 172.21.15.99:0/2519223484' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b377d95f-37bd-44b3-a231-13d69245430a"}]': finished 2024-01-15T22:28:30.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:30 smithi099 conmon[35292]: cluster 2024-01-15T22:28:29.502492+0000 mon.smithi099 (mon.0) 219 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-01-15T22:28:30.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:30 smithi099 conmon[35292]: audit 2024-01-15T22:28:29.502577+0000 mon.smithi099 (mon.0) 220 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-15T22:28:30.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:30 smithi099 conmon[35292]: audit 2024-01-15T22:28:29.554396+0000 mon.smithi162 (mon.1) 2 : audit [INF] from='client.? 172.21.15.162:0/671559912' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72af62ca-597d-4028-a593-755323626ad5"}]: dispatch 2024-01-15T22:28:30.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:30 smithi099 conmon[35292]: audit 2024-01-15T22:28:29 2024-01-15T22:28:30.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:30 smithi099 conmon[35292]: .554755+0000 mon.smithi099 (mon.0) 221 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72af62ca-597d-4028-a593-755323626ad5"}]: dispatch 2024-01-15T22:28:30.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:30 smithi099 conmon[35292]: audit 2024-01-15T22:28:29.562650+0000 mon.smithi099 (mon.0) 222 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "72af62ca-597d-4028-a593-755323626ad5"}]': finished 2024-01-15T22:28:30.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:30 smithi099 conmon[35292]: cluster 2024-01-15T22:28:29.562699+0000 mon.smithi099 (mon.0) 223 : cluster [DBG] osdmap e7: 2 total, 0 up, 2 in 2024-01-15T22:28:30.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:30 smithi099 conmon[35292]: audit 2024-01-15T22:28:29.562823+0000 mon.smithi099 (mon.0) 224 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-15T22:28:30.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:30 smithi099 conmon[35292]: audit 2024-01-15T22:28:29.563065+0000 mon.smithi099 (mon.0) 225 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-15T22:28:30.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:30 smithi099 conmon[35292]: audit 2024-01-15T22:28:29.834001+0000 mon.smithi099 (mon.0) 226 : audit [INF] from='client.? 172.21.15.99:0/1541060609' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:28:30.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:30 smithi099 conmon[35292]: audit 2024-01-15T22:28:29.842774+0000 mon.smithi099 (mon.0) 227 : audit [INF] from='client.? 172.21.15.99:0/1541060609' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-15T22:28:30.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:30 smithi099 conmon[35292]: cluster 2024-01-15T22:28:29.842849+0000 mon.smithi099 (mon.0) 228 : cluster [DBG] osdmap e8: 1 total, 0 up, 1 in 2024-01-15T22:28:30.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:30 smithi099 conmon[35292]: audit 2024-01-15T22:28:29.842953+0000 mon.smithi099 (mon.0) 229 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-15T22:28:30.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:30 smithi099 conmon[35292]: audit 2024-01-15T22:28:29.850257+0000 mon.smithi162 (mon.1) 3 : audit [INF] from='client.? 172.21.15.162:0/3442818419' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:28:30.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:30 smithi099 conmon[35292]: audit 2024-01-15T22:28:29.850613+0000 mon.smithi099 (mon.0) 230 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:28:30.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:30 smithi099 conmon[35292]: audit 2024-01-15T22:28:29.859546+0000 mon.smithi099 (mon.0) 231 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]': finished 2024-01-15T22:28:30.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:30 smithi099 conmon[35292]: cluster 2024-01-15T22:28:29.859628+0000 mon.smithi099 (mon.0) 232 : cluster [DBG] osdmap e9: 0 total, 0 up, 0 in 2024-01-15T22:28:30.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:30 smithi099 conmon[35292]: audit 2024-01-15T22:28:29.906983+0000 mon.smithi099 (mon.0) 233 : audit [DBG] from='client.? 172.21.15.99:0/10245631' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:28:30.659 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:30 smithi162 conmon[39755]: cluster 2024-01-15T22:28:29.207409+0000 mgr.smithi099.cmjeyn (mgr.14184) 30 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:30.659 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:30 smithi162 conmon[39755]: audit 2024-01-15T22 2024-01-15T22:28:30.659 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:30 smithi162 conmon[39755]: :28:29.497000+0000 mon.smithi099 (mon.0) 217 : audit [INF] from='client.? 172.21.15.99:0/2519223484' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b377d95f-37bd-44b3-a231-13d69245430a"}]: dispatch 2024-01-15T22:28:30.659 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:30 smithi162 conmon[39755]: audit 2024-01-15T22:28:29.502453+0000 mon.smithi099 (mon.0) 218 : audit [INF] from='client.? 172.21.15.99:0/2519223484' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b377d95f-37bd-44b3-a231-13d69245430a"}]': finished 2024-01-15T22:28:30.659 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:30 smithi162 conmon[39755]: cluster 2024-01-15T22:28:29.502492+0000 mon.smithi099 (mon.0) 219 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-01-15T22:28:30.659 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:30 smithi162 conmon[39755]: audit 2024-01-15T22:28:29.502577+0000 mon.smithi099 (mon.0) 220 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-15T22:28:30.659 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:30 smithi162 conmon[39755]: audit 2024-01-15T22:28:29.554396+0000 mon.smithi162 (mon.1) 2 : audit [INF] from='client.? 172.21.15.162:0/671559912' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72af62ca-597d-4028-a593-755323626ad5"}]: dispatch 2024-01-15T22:28:30.659 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:30 smithi162 conmon[39755]: audit 2024-01-15T22:28:29.554755+0000 mon.smithi099 (mon.0) 221 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72af62ca-597d-4028-a593-755323626ad5"}]: dispatch 2024-01-15T22:28:30.659 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:30 smithi162 conmon[39755]: audit 2024-01-15T22:28:29.562650+0000 mon.smithi099 (mon.0) 222 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "72af62ca-597d-4028-a593-755323626ad5"}]': finished 2024-01-15T22:28:30.659 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:30 smithi162 conmon[39755]: cluster 2024-01-15T22:28:29.562699+0000 mon.smithi099 (mon.0) 223 : cluster [DBG] osdmap e7: 2 total, 0 up, 2 in 2024-01-15T22:28:30.659 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:30 smithi162 conmon[39755]: audit 2024-01-15T22:28:29.562823+0000 mon.smithi099 (mon.0) 224 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-15T22:28:30.659 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:30 smithi162 conmon[39755]: audit 2024-01-15T22:28:29.563065+0000 mon.smithi099 (mon.0) 225 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-15T22:28:30.660 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:30 smithi162 conmon[39755]: audit 2024-01-15T22:28:29.834001+0000 mon.smithi099 (mon.0) 226 : audit [INF] from='client.? 172.21.15.99:0/1541060609' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:28:30.660 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:30 smithi162 conmon[39755]: audit 2024-01-15T22:28:29.842774+0000 mon.smithi099 (mon.0) 227 : audit [INF] from='client.? 172.21.15.99:0/1541060609' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-15T22:28:30.660 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:30 smithi162 conmon[39755]: cluster 2024-01-15T22:28:29.842849+0000 mon.smithi099 (mon.0) 228 : cluster [DBG] osdmap e8: 1 total, 0 up, 1 in 2024-01-15T22:28:30.660 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:30 smithi162 conmon[39755]: audit 2024-01-15T22:28:29.842953+0000 mon.smithi099 (mon.0) 229 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-15T22:28:30.660 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:30 smithi162 conmon[39755]: audit 2024-01-15T22:28:29.850257+0000 mon.smithi162 (mon.1) 3 : audit [INF] from='client.? 172.21.15.162:0/3442818419' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:28:30.660 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:30 smithi162 conmon[39755]: audit 2024-01-15T22:28:29.850613+0000 mon.smithi099 (mon.0) 230 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:28:30.660 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:30 smithi162 conmon[39755]: audit 2024-01-15T22:28:29.859546+0000 mon.smithi099 (mon.0) 231 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]': finished 2024-01-15T22:28:30.660 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:30 smithi162 conmon[39755]: cluster 2024-01-15T22:28:29.859628+0000 mon.smithi099 (mon.0) 232 : cluster [DBG] osdmap e9: 0 total, 0 up, 0 in 2024-01-15T22:28:30.660 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:30 smithi162 conmon[39755]: audit 2024-01-15T22:28:29.906983+0000 mon.smithi099 (mon.0) 233 : audit [DBG] from='client.? 172.21.15.99:0/10245631' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:28:30.799 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":9,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:28:31.632 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: cephadm 2024-01-15T22:28:30.682158+0000 mgr.smithi099.cmjeyn (mgr.14184) 31 : cephadm 2024-01-15T22:28:31.632 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi162', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-15T22:28:31.632 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:28:31.632 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:28:31.632 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: return f(*arg) 2024-01-15T22:28:31.632 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:28:31.633 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:28:31.633 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:28:31.633 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: code, '\n'.join(err))) 2024-01-15T22:28:31.633 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:28:31.633 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpu35owhk0:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpkyyupn14:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:28:31.633 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:28:31.633 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:28:31.633 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:28:31.633 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 72af62ca-597d-4028-a593-755323626ad5 2024-01-15T22:28:31.633 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:28:31.633 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 --yes-i-really-mean-it 2024-01-15T22:28:31.633 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr stderr: purged osd.1 2024-01-15T22:28:31.633 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:28:31.634 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:28:31.634 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr self.prepare() 2024-01-15T22:28:31.634 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:31.634 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:31.634 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:28:31.634 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:28:31.634 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:28:31.634 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:28:31.634 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:28:31.634 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:28:31.634 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:28:31.634 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:28:31.634 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:28:31.634 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:28:31.634 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:28:31.635 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:28:31.635 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:28:31.635 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:28:31.635 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:28:31.635 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:28:31.635 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:28:31.635 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:28:31.635 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:28:31.635 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:28:31.635 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:28:31.635 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:28:31.635 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:28:31.635 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:28:31.635 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:28:31.636 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:31.636 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:31.636 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:28:31.636 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:28:31.636 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:28:31.636 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:28:31.636 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:31.636 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:31.636 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:28:31.636 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:28:31.636 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:28:31.636 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:28:31.636 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:28:31.636 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:28:31.637 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:28:31.637 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:28:31.637 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:31.637 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:31.637 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:28:31.637 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:28:31.637 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:28:31.637 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:28:31.637 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 1 2024-01-15T22:28:31.637 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:28:31.637 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:28:31.637 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: main() 2024-01-15T22:28:31.637 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:28:31.638 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: r = ctx.func(ctx) 2024-01-15T22:28:31.638 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:28:31.638 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:28:31.638 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:28:31.638 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:28:31.638 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:28:31.638 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:28:31.638 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", lin 2024-01-15T22:28:31.638 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: e 1927, in _validate_fsid 2024-01-15T22:28:31.638 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:28:31.638 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:28:31.638 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:28:31.638 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:28:31.638 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:28:31.638 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpu35owhk0:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpkyyupn14:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:28:31.639 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: cephadm 2024-01-15T22:28:30.772462+0000 mgr.smithi099.cmjeyn (mgr.14184) 32 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi162', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-15T22:28:31.639 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:28:31.639 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:28:31.639 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: return f(*arg) 2024-01-15T22:28:31.639 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:28:31.639 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:28:31.639 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:28:31.639 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: code, '\n'.join(err))) 2024-01-15T22:28:31.639 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi099/config 2024-01-15T22:28:31.639 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpdxjhbyq5:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp0l9842b6:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:28:31.639 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:28:31.639 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:28:31.639 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:28:31.639 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new b377d95f-37bd-44b3-a231-13d69245430a 2024-01-15T22:28:31.640 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:28:31.640 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:28:31.640 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:28:31.640 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:28:31.640 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:28:31.640 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr self.prepare() 2024-01-15T22:28:31.640 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:31.640 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:31.640 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:28:31.640 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:28:31.640 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:28:31.640 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:28:31.640 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:28:31.641 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:28:31.641 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:28:31.641 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:28:31.641 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:28:31.641 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:28:31.641 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:28:31.641 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:28:31.641 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:28:31.641 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:28:31.641 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:28:31.641 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:28:31.641 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:28:31.641 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:28:31.641 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:28:31.641 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:28:31.642 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:28:31.642 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:28:31.642 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:28:31.642 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:28:31.642 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:28:31.642 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:31.642 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:31.642 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:28:31.642 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:28:31.642 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:28:31.642 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:28:31.642 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:31.642 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:31.642 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:28:31.642 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:28:31.643 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:28:31.643 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:28:31.643 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:28:31.643 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:28:31.643 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: 2024-01-15T22:28:31.643 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:28:31.643 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:28:31.643 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:31.643 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:31.643 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:28:31.643 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:28:31.643 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:28:31.643 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:28:31.643 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:28:31.643 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:28:31.644 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:28:31.644 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: main() 2024-01-15T22:28:31.644 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:28:31.644 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: r = ctx.func(ctx) 2024-01-15T22:28:31.645 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:28:31.645 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:28:31.645 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:28:31.645 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:28:31.645 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:28:31.645 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:28:31.645 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:28:31.645 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:28:31.646 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:28:31.646 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:28:31.646 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:28:31.646 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:28:31.646 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpdxjhbyq5:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp0l9842b6:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:28:31.646 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: cephadm 2024-01-15T22:28:30.774057+0000 mgr.smithi099.cmjeyn (mgr.14184) 33 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-15T22:28:31.646 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: service_id: all-available-devices 2024-01-15T22:28:31.646 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: service_name: osd.all-available-devices 2024-01-15T22:28:31.646 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: placement: 2024-01-15T22:28:31.646 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: host_pattern: '*' 2024-01-15T22:28:31.646 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: spec: 2024-01-15T22:28:31.646 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: data_devices: 2024-01-15T22:28:31.646 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: all: true 2024-01-15T22:28:31.646 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: filter_logic: AND 2024-01-15T22:28:31.646 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: objectstore: bluestore 2024-01-15T22:28:31.646 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:28:31.647 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpu35owhk0:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpkyyupn14:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:28:31.647 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:28:31.647 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:28:31.647 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:28:31.647 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 72af62ca-597d-4028-a593-755323626ad5 2024-01-15T22:28:31.647 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:28:31.647 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 --yes-i-really-mean-it 2024-01-15T22:28:31.647 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr stderr: purged osd.1 2024-01-15T22:28:31.647 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:28:31.647 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:28:31.647 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr self.prepare() 2024-01-15T22:28:31.647 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:31.647 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:31.647 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:28:31.648 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:28:31.648 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:28:31.648 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:28:31.648 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:28:31.648 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:28:31.648 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:28:31.648 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:28:31.648 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:28:31.648 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:28:31.648 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:28:31.648 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:28:31.648 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:28:31.648 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:28:31.648 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:28:31.649 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:28:31.649 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:28:31.649 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:28:31.649 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:28:31.649 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:28:31.649 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:28:31.649 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:28:31.649 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv 2024-01-15T22:28:31.649 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: ) 2024-01-15T22:28:31.649 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:28:31.649 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:28:31.649 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:31.649 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:31.649 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:28:31.649 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:28:31.650 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:28:31.650 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:28:31.650 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:31.650 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:31.650 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:28:31.650 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:28:31.650 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:28:31.650 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:28:31.650 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:28:31.650 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:28:31.650 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:28:31.650 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:28:31.650 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:31.650 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:31.650 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:28:31.650 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:28:31.651 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:28:31.651 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:28:31.651 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 1 2024-01-15T22:28:31.651 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:28:31.651 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:28:31.651 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: main() 2024-01-15T22:28:31.651 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:28:31.651 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: r = ctx.func(ctx) 2024-01-15T22:28:31.651 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:28:31.651 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:28:31.651 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:28:31.651 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:28:31.651 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:28:31.651 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:28:31.652 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:28:31.652 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:28:31.652 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:28:31.652 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:28:31.652 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:28:31.652 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:28:31.652 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpu35owhk0:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpkyyupn14:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:28:31.652 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:28:31.652 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-15T22:28:31.652 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: if self._apply_service(spec): 2024-01-15T22:28:31.652 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-15T22:28:31.652 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-15T22:28:31.652 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-15T22:28:31.652 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-15T22:28:31.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-15T22:28:31.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-15T22:28:31.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-15T22:28:31.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-15T22:28:31.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-15T22:28:31.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: raise self._value 2024-01-15T22:28:31.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-15T22:28:31.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: result = (True, func(*args, **kwds)) 2024-01-15T22:28:31.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-15T22:28:31.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: return list(map(*args)) 2024-01-15T22:28:31.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:28:31.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: return f(*arg) 2024-01-15T22:28:31.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:28:31.653 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:28:31.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:28:31.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: code, '\n'.join(err))) 2024-01-15T22:28:31.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:28:31.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpu35owhk0:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpkyyupn14:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:28:31.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:28:31.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:28:31.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:28:31.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 72af62ca-597d-4028-a593-755323626ad5 2024-01-15T22:28:31.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:28:31.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr 2024-01-15T22:28:31.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 --yes-i-really-mean-it 2024-01-15T22:28:31.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr stderr: purged osd.1 2024-01-15T22:28:31.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:28:31.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:28:31.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr self.prepare() 2024-01-15T22:28:31.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:31.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:31.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:28:31.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:28:31.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:28:31.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:28:31.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:28:31.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:28:31.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:28:31.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:28:31.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:28:31.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:28:31.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:28:31.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:28:31.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:28:31.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:28:31.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:28:31.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:28:31.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:28:31.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:28:31.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:28:31.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:28:31.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:28:31.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:28:31.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:28:31.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:28:31.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:28:31.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:31.657 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:31.657 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:28:31.657 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:28:31.657 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:28:31.657 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:28:31.657 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:31.657 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:31.657 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:28:31.657 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:28:31.657 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:28:31.657 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:28:31.657 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:28:31.657 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:28:31.657 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:28:31.657 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:28:31.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:31.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:31.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:28:31.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:28:31.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:28:31.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:28:31.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 1 2024-01-15T22:28:31.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:28:31.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:28:31.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: main() 2024-01-15T22:28:31.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:28:31.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: r = ctx.func(ctx) 2024-01-15T22:28:31.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:28:31.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:28:31.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:28:31.659 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:28:31.659 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:28:31.659 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:28:31.659 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:28:31.659 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:28:31.659 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:28:31.659 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:28:31.659 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:28:31.659 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:28:31.659 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpu35owhk0:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpkyyupn14:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:28:31.659 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: cephadm 2024-01-15T22:28:30.774537+0000 mgr.smithi099.cmjeyn (mgr.14184) 34 : cephadm [INF] Reconfiguring mon.smithi099 (unknown last config time)... 2024-01-15T22:28:31.659 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: cluster 2024-01-15T22:28:30.774741+0000 mgr.smithi099.cmjeyn (mgr.14184) 35 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:31.659 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: audit 2024-01-15T22:28:30.774910+0000 mon.smithi099 (mon.0) 234 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-15T22:28:31.659 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: audit 2024-01-15T22:28:30.775694+0000 mon.smithi099 (mon.0) 235 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-15T22:28:31.659 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: audit 2024-01-15T22:28:30.776259+0000 mon.smithi099 (mon.0) 236 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config 2024-01-15T22:28:31.660 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: generate-minimal-conf"}]: dispatch 2024-01-15T22:28:31.660 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:31 smithi099 conmon[35292]: cephadm 2024-01-15T22:28:30.776829+0000 mgr.smithi099.cmjeyn (mgr.14184) 36 : cephadm [INF] Reconfiguring daemon mon.smithi099 on smithi099 2024-01-15T22:28:31.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: cephadm 2024-01-15T22:28:30.682158+0000 mgr.smithi099.cmjeyn (mgr.14184) 31 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi162', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-15T22:28:31.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:28:31.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:28:31.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: return f(*arg) 2024-01-15T22:28:31.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:28:31.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:28:31.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:28:31.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: code, '\n'.join(err))) 2024-01-15T22:28:31.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:28:31.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpu35owhk0:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpkyyupn14:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:28:31.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:28:31.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:28:31.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:28:31.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 72af62ca-597d-4028-a593-755323626ad5 2024-01-15T22:28:31.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:28:31.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 --yes-i-really-mean-it 2024-01-15T22:28:31.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr stderr: purged osd.1 2024-01-15T22:28:31.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:28:31.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:28:31.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr self.prepare() 2024-01-15T22:28:31.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:31.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:31.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:28:31.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:28:31.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:28:31.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:28:31.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:28:31.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:28:31.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:28:31.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:28:31.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:28:31.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:28:31.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:28:31.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:28:31.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:28:31.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:28:31.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:28:31.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:28:31.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:28:31.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:28:31.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:28:31.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:28:31.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:28:31.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:28:31.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:28:31.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:28:31.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:28:31.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:31.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:31.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:28:31.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:28:31.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:28:31.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:28:31.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:31.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:31.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:28:31.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:28:31.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:28:31.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:28:31.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:28:31.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:28:31.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:28:31.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:28:31.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:31.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:31.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:28:31.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:28:31.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:28:31.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:28:31.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 1 2024-01-15T22:28:31.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:28:31.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:28:31.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: main() 2024-01-15T22:28:31.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:28:31.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: r = ctx.func(ctx) 2024-01-15T22:28:31.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:28:31.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:28:31.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:28:31.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:28:31.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:28:31.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:28:31.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774 2024-01-15T22:28:31.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:28:31.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:28:31.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:28:31.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:28:31.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:28:31.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:28:31.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpu35owhk0:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpkyyupn14:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:28:31.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: cephadm 2024-01-15T22:28:30.772462+0000 mgr.smithi099.cmjeyn (mgr.14184) 32 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi162', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-15T22:28:31.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:28:31.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:28:31.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: return f(*arg) 2024-01-15T22:28:31.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:28:31.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:28:31.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:28:31.704 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: code, '\n'.join(err))) 2024-01-15T22:28:31.704 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi099/config 2024-01-15T22:28:31.704 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpdxjhbyq5:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp0l9842b6:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:28:31.705 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:28:31.705 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:28:31.705 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:28:31.705 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new b377d95f-37bd-44b3-a231-13d69245430a 2024-01-15T22:28:31.705 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:28:31.705 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:28:31.705 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:28:31.706 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:28:31.706 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:28:31.706 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr self.prepare() 2024-01-15T22:28:31.706 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:31.706 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:31.706 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:28:31.706 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:28:31.706 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:28:31.706 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:28:31.706 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:28:31.706 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:28:31.706 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:28:31.706 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:28:31.706 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:28:31.706 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:28:31.706 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:28:31.707 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:28:31.707 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:28:31.707 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:28:31.707 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:28:31.707 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:28:31.707 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:28:31.707 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:28:31.707 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:28:31.707 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:28:31.707 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:28:31.707 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:28:31.707 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:28:31.707 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:28:31.707 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:28:31.707 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:31.707 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:31.708 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:28:31.708 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:28:31.708 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:28:31.708 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:28:31.708 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:31.708 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:31.708 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:28:31.708 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:28:31.708 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:28:31.708 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:28:31.708 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_ 2024-01-15T22:28:31.708 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: osd 2024-01-15T22:28:31.708 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:28:31.708 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:28:31.709 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:28:31.709 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:31.709 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:31.709 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:28:31.709 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:28:31.709 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:28:31.709 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:28:31.709 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:28:31.709 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:28:31.709 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:28:31.709 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: main() 2024-01-15T22:28:31.709 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:28:31.709 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: r = ctx.func(ctx) 2024-01-15T22:28:31.709 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:28:31.709 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:28:31.709 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:28:31.710 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:28:31.710 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:28:31.710 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:28:31.710 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:28:31.710 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:28:31.710 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:28:31.710 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:28:31.710 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:28:31.710 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:28:31.710 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpdxjhbyq5:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp0l9842b6:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:28:31.710 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: cephadm 2024-01-15T22:28:30.774057+0000 mgr.smithi099.cmjeyn (mgr.14184) 33 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-15T22:28:31.710 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: service_id: all-available-devices 2024-01-15T22:28:31.710 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: service_name: osd.all-available-devices 2024-01-15T22:28:31.710 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: placement: 2024-01-15T22:28:31.711 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: host_pattern: '*' 2024-01-15T22:28:31.711 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: spec: 2024-01-15T22:28:31.711 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: data_devices: 2024-01-15T22:28:31.711 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: all: true 2024-01-15T22:28:31.711 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: filter_logic: AND 2024-01-15T22:28:31.711 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: objectstore: bluestore 2024-01-15T22:28:31.711 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:28:31.711 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpu35owhk0:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpkyyupn14:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:28:31.711 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:28:31.711 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:28:31.711 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:28:31.711 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 72af62ca-597d-4028-a593-755323626ad5 2024-01-15T22:28:31.711 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:28:31.711 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 --yes-i-really-mean-it 2024-01-15T22:28:31.711 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr stderr: purged osd.1 2024-01-15T22:28:31.712 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:28:31.712 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:28:31.712 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr self.prepare() 2024-01-15T22:28:31.712 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:31.712 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:31.712 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:28:31.712 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:28:31.712 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:28:31.712 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:28:31.712 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:28:31.712 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:28:31.712 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:28:31.712 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:28:31.712 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:28:31.713 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:28:31.713 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:28:31.713 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:28:31.713 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:28:31.713 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:28:31.713 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:28:31.713 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:28:31.713 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:28:31.713 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:28:31.713 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:28:31.713 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:28:31.713 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:28:31.713 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main. 2024-01-15T22:28:31.713 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: py", line 46, in main 2024-01-15T22:28:31.713 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:28:31.714 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:28:31.714 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:28:31.714 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:31.714 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:31.714 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:28:31.714 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:28:31.714 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:28:31.714 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:28:31.714 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:31.714 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:31.714 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:28:31.714 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:28:31.714 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:28:31.714 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:28:31.714 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:28:31.714 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:28:31.715 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:28:31.715 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:28:31.715 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:31.715 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:31.715 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:28:31.715 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:28:31.715 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:28:31.715 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:28:31.715 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 1 2024-01-15T22:28:31.715 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:28:31.715 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:28:31.715 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: main() 2024-01-15T22:28:31.715 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:28:31.715 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: r = ctx.func(ctx) 2024-01-15T22:28:31.716 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:28:31.716 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:28:31.716 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:28:31.716 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:28:31.716 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:28:31.716 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:28:31.716 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:28:31.716 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:28:31.716 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:28:31.716 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:28:31.716 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:28:31.716 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:28:31.716 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpu35owhk0:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpkyyupn14:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:28:31.716 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:28:31.717 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-15T22:28:31.717 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: if self._apply_service(spec): 2024-01-15T22:28:31.717 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-15T22:28:31.717 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-15T22:28:31.717 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-15T22:28:31.717 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-15T22:28:31.717 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-15T22:28:31.717 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-15T22:28:31.717 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-15T22:28:31.717 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-15T22:28:31.717 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-15T22:28:31.717 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: raise self._value 2024-01-15T22:28:31.717 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-15T22:28:31.717 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: result = (True, func(*args, **kwds)) 2024-01-15T22:28:31.717 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-15T22:28:31.718 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: return list(map(*args)) 2024-01-15T22:28:31.718 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:28:31.718 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: return f(*arg) 2024-01-15T22:28:31.718 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:28:31.718 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:28:31.718 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:28:31.718 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: code, '\n'.join(err))) 2024-01-15T22:28:31.718 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:28:31.718 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpu35owhk0:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpkyyupn14:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:28:31.718 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:28:31.718 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:28:31.718 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:28:31.718 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 72af62ca-597d-4028-a593-755323626ad5 2024-01-15T22:28:31.718 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr --> Was unable 2024-01-15T22:28:31.718 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: to complete a new OSD, will rollback changes 2024-01-15T22:28:31.718 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 --yes-i-really-mean-it 2024-01-15T22:28:31.719 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr stderr: purged osd.1 2024-01-15T22:28:31.719 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:28:31.719 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:28:31.719 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr self.prepare() 2024-01-15T22:28:31.719 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:31.719 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:31.719 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:28:31.719 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:28:31.719 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:28:31.719 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:28:31.719 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:28:31.719 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:28:31.719 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:28:31.719 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:28:31.719 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:28:31.719 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:28:31.720 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:28:31.720 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:28:31.720 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:28:31.720 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:28:31.720 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:28:31.720 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:28:31.720 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:28:31.720 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:28:31.720 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:28:31.720 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:28:31.720 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:28:31.720 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:28:31.720 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:28:31.720 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:28:31.720 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:28:31.721 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:31.721 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:31.721 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:28:31.721 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:28:31.721 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:28:31.721 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:28:31.721 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:31.721 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:31.721 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:28:31.721 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:28:31.721 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:28:31.721 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:28:31.721 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:28:31.721 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:28:31.721 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:28:31.722 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:28:31.722 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:31.722 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:31.722 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:28:31.722 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:28:31.722 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:28:31.722 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:28:31.722 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 1 2024-01-15T22:28:31.722 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:28:31.722 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:28:31.722 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: main() 2024-01-15T22:28:31.722 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:28:31.722 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: r = ctx.func(ctx) 2024-01-15T22:28:31.722 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:28:31.722 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:28:31.723 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:28:31.723 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:28:31.723 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:28:31.723 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:28:31.723 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:28:31.723 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:28:31.723 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:28:31.723 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:28:31.723 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:28:31.723 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:28:31.723 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpu35owhk0:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpkyyupn14:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:28:31.723 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: cephadm 2024-01-15T22:28:30.774537+0000 mgr.smithi099.cmjeyn (mgr.14184) 34 : cephadm [INF] Reconfiguring mon.smithi099 (unknown last config time)... 2024-01-15T22:28:31.723 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: cluster 2024-01-15T22:28:30.774741+0000 mgr.smithi099.cmjeyn (mgr.14184) 35 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:31.723 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: audit 2024-01-15T22:28:30.774910+0000 mon.smithi099 (mon.0) 234 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-15T22:28:31.724 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: audit 2024-01-15T22:28:30.775694+0000 mon.smithi099 (mon.0) 235 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-15T22:28:31.724 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: audit 2024-01-15T22:28:30.776259+0000 mon.smithi099 (mon.0) 236 : audit [DBG] from='mgr.1 2024-01-15T22:28:31.724 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: 4184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:28:31.724 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:31 smithi162 conmon[39755]: cephadm 2024-01-15T22:28:30.776829+0000 mgr.smithi099.cmjeyn (mgr.14184) 36 : cephadm [INF] Reconfiguring daemon mon.smithi099 on smithi099 2024-01-15T22:28:31.801 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:28:32.280 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:32 smithi099 conmon[35292]: cluster 2024-01-15T22:28:31.260738+0000 mon.smithi099 (mon 2024-01-15T22:28:32.280 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:32 smithi099 conmon[35292]: .0) 237 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-15T22:28:32.280 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:32 smithi099 conmon[35292]: audit 2024-01-15T22:28:32.068885+0000 mon.smithi099 (mon.0) 238 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:32.280 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:32 smithi099 conmon[35292]: audit 2024-01-15T22:28:32.069799+0000 mon.smithi099 (mon.0) 239 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi099.cmjeyn", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-15T22:28:32.280 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:32 smithi099 conmon[35292]: audit 2024-01-15T22:28:32.070487+0000 mon.smithi099 (mon.0) 240 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-15T22:28:32.281 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:32 smithi099 conmon[35292]: audit 2024-01-15T22:28:32.071008+0000 mon.smithi099 (mon.0) 241 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:28:32.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:32 smithi162 conmon[39755]: cluster 2024-01-15T22:28:31.260738+0000 mon.smithi099 (mon.0) 237 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-15T22:28:32.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:32 smithi162 conmon[39755]: audit 2024-01-15T22:28:32.068885+0000 mon.smithi099 (mon.0) 238 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:32.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:32 smithi162 conmon[39755]: audit 2024-01-15T22:28:32.069799+0000 mon.smithi099 (mon.0) 239 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi099.cmjeyn", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-15T22:28:32.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:32 smithi162 conmon[39755]: audit 2024-01-15T22:28:32.070487+0000 mon.smithi099 (mon.0) 240 : audit [DBG] 2024-01-15T22:28:32.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:32 smithi162 conmon[39755]: from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-15T22:28:32.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:32 smithi162 conmon[39755]: audit 2024 2024-01-15T22:28:32.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:32 smithi162 conmon[39755]: -01-15T22:28:32.071008+0000 mon.smithi099 (mon.0) 241 2024-01-15T22:28:32.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:32 smithi162 conmon[39755]: : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:28:33.605 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:33 smithi099 conmon[35292]: cephadm 2024-01-15T22:28:32.069554+0000 mgr.smithi099.cmjeyn (mgr.14184) 37 : cephadm [INF] Reconfiguring mgr.smithi099.cmjeyn (unknown last config time)... 2024-01-15T22:28:33.605 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:33 smithi099 conmon[35292]: cephadm 2024-01-15T22:28:32.071600+0000 mgr.smithi099.cmjeyn (mgr.14184) 38 : cephadm [INF] Reconfiguring daemon mgr.smithi099.cmjeyn on smithi099 2024-01-15T22:28:33.605 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:33 smithi099 conmon[35292]: cluster 2024-01-15T22:28:32.775355+0000 mgr.smithi099.cmjeyn (mgr.14184) 39 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:33.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:33 smithi162 conmon[39755]: cephadm 2024-01-15T22:28:32.069554+0000 mgr.smithi099.cmjeyn (mgr.14184) 37 : cephadm [INF] Reconfiguring mgr.smithi099.cmjeyn (unknown last config time)... 2024-01-15T22:28:33.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:33 smithi162 conmon[39755]: cephadm 2024- 2024-01-15T22:28:33.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:33 smithi162 conmon[39755]: 01-15T22:28:32.071600+0000 mgr.smithi099.cmjeyn (mgr.14184) 38 : cephadm [INF] Reconfiguring daemon mgr.smithi099.cmjeyn on smithi099 2024-01-15T22:28:33.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:33 smithi162 conmon[39755]: cluster 2024-01-15T22:28:32.775355+0000 mgr.smithi099.cmjeyn (mgr.14184) 39 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:34.156 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:28:34.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:34 smithi099 conmon[35292]: audit 2024-01-15T22:28:33.611035+0000 2024-01-15T22:28:34.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:34 smithi099 conmon[35292]: mon.smithi099 (mon.0) 242 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:34.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:34 smithi099 conmon[35292]: cephadm 2024-01-15T22:28:33.611914+0000 mgr.smithi099.cmjeyn (mgr.14184) 40 : cephadm [INF] Reconfiguring alertmanager.smithi099 (dependencies changed)... 2024-01-15T22:28:34.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:34 smithi099 conmon[35292]: cephadm 2024-01-15T22:28:33.620284+0000 mgr.smithi099.cmjeyn (mgr.14184) 41 : cephadm [INF] Reconfiguring daemon alertmanager.smithi099 on smithi099 2024-01-15T22:28:34.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:34 smithi099 conmon[35292]: audit 2024-01-15T22:28:34.157063+0000 mon.smithi099 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.99:0/3479664223' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:28:34.946 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":9,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:28:34.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:34 smithi162 conmon[39755]: audit 2024-01-15T22:28:33.611035+0000 mon.smithi099 (mon.0) 242 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:34.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:34 smithi162 conmon[39755]: cephadm 2024-01-15T22:28:33.611914+0000 mgr.smithi099.cmjeyn (mgr.14184) 40 : cephadm [INF] Reconfiguring alertmanager.smithi099 (dependencies changed)... 2024-01-15T22:28:34.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:34 smithi162 conmon[39755]: cephadm 2024-01-15T22:28:33.620284+0000 mgr.smithi099.cmjeyn (mgr. 2024-01-15T22:28:34.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:34 smithi162 conmon[39755]: 14184) 41 : cephadm [INF] Reconfiguring daemon alertmanager.smithi099 on smithi099 2024-01-15T22:28:34.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:34 smithi162 conmon[39755]: audit 2024-01-15T22: 2024-01-15T22:28:34.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:34 smithi162 conmon[39755]: 28:34.157063+0000 mon.smithi099 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.99:0/3479664223' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:28:35.633 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:35 smithi099 conmon[35292]: cluster 2024-01-15T22:28:34.775955+0000 mgr.smithi099.cmjeyn (mgr.14184) 42 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:35.946 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:35 smithi162 conmon[39755]: cluster 2024-01-15T22:28:34.775955+0000 mgr.smithi099.cmjeyn (mgr.14184) 42 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:35.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:35 smithi162 conmon[39755]: 2024-01-15T22:28:35.947 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:28:37.098 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:36 smithi099 conmon[35292]: cluster 2024-01-15T22:28:36.776309+0000 mgr.smithi099.cmjeyn (mgr.14184) 43 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:37.196 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:36 smithi162 conmon[39755]: cluster 2024-01-15T22:28:36.776309+0000 mgr.smithi099.cmjeyn (mgr.14184) 43 : 2024-01-15T22:28:37.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:36 smithi162 conmon[39755]: cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:38.425 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:28:38.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:38 smithi099 conmon[35292]: audit 2024-01-15T22:28:38.424719+0000 mon.smithi099 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.99:0/1536287116' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:28:38.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:38 smithi162 conmon[39755]: audit 2024-01-15T22:28:38.424719+0000 mon.smithi099 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.99:0/1536287116' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:28:39.253 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":9,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:28:39.801 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:39 smithi099 conmon[35292]: cluster 2024-01-15T22:28:38.776979+0000 mgr.smithi099.cmjeyn (mgr.14184) 44 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:39.801 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:39 smithi099 conmon[35292]: audit 2024-01-15T22:28:39.217727+0000 mon.smithi099 (mon.0) 245 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:39.801 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:39 smithi099 conmon[35292]: audit 2024-01-15T22:28:39.218507+0000 mon.smithi099 (mon.0) 246 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi099", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-15T22:28:39.801 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:39 smithi099 conmon[35292]: audit 2024-01-15T22:28:39.219106+0000 mon.smithi099 (mon.0) 247 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:28:39.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:39 smithi162 conmon[39755]: cluster 2024-01-15T22:28:38.776979+0000 mgr.smithi099.cmjeyn (mgr.14184) 44 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:39.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:39 smithi162 conmon[39755]: audit 2024-01-15T22:28:39.217727+0000 mon.smithi099 (mon.0) 245 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:39.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:39 smithi162 conmon[39755]: audit 2024-01-15T22:28:39.218507+0000 mon.smithi099 (mon. 2024-01-15T22:28:39.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:39 smithi162 conmon[39755]: 0) 246 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi099", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-15T22:28:39.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:39 smithi162 conmon[39755]: audit 2024-01-15T22:28:39.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:39 smithi162 conmon[39755]: 2024-01-15T22:28:39.219106+0000 mon.smithi099 (mon.0) 247 : 2024-01-15T22:28:39.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:39 smithi162 conmon[39755]: audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:28:40.255 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:28:40.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:40 smithi099 conmon[35292]: cephadm 2024-01-15T22:28:39.218276+0000 mgr.smithi099.cmjeyn (mgr.14184) 45 : cephadm [INF] Reconfiguring crash.smithi099 (monmap changed)... 2024-01-15T22:28:40.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:40 smithi099 conmon[35292]: cephadm 2024-01-15T22:28:39.219553+0000 mgr.smithi099.cmjeyn (mgr.14184) 46 : cephadm [INF] Reconfiguring daemon crash.smithi099 on smithi099 2024-01-15T22:28:40.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:40 smithi099 conmon[35292]: audit 2024-01-15T22:28:40.386478+0000 mon.smithi099 (mon.0) 248 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:40.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:40 smithi162 conmon[39755]: cephadm 2024-01-15T22:28:39.218276+0000 mgr.smithi099.cmjeyn (mgr.14184) 45 : cephadm [INF] Reconfiguring crash.smithi099 (monmap changed)... 2024-01-15T22:28:40.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:40 smithi162 conmon[39755]: 2024-01-15T22:28:40.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:40 smithi162 conmon[39755]: cephadm 2024-01-15T22:28:39.219553+0000 mgr.smithi099.cmjeyn (mgr.14184) 46 : cephadm [INF] Reconfiguring daemon crash.smithi099 on smithi099 2024-01-15T22:28:40.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:40 smithi162 conmon[39755]: audit 2024-01-15T22:28:40. 2024-01-15T22:28:40.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:40 smithi162 conmon[39755]: 386478+0000 mon.smithi099 (mon.0) 248 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:41.830 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:41 smithi099 conmon[35292]: cephadm 2024-01-15T22:28:40.387086+0000 mgr.smithi099.cmjeyn (mgr.14184) 47 : 2024-01-15T22:28:41.830 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:41 smithi099 conmon[35292]: cephadm [INF] Reconfiguring grafana.smithi099 (dependencies changed)... 2024-01-15T22:28:41.830 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:41 smithi099 conmon[35292]: cephadm 2024-01-15T22:28:40.419076+0000 mgr.smithi099.cmjeyn (mgr.14184) 48 : cephadm [INF] Reconfiguring daemon grafana.smithi099 on smithi099 2024-01-15T22:28:41.830 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:41 smithi099 conmon[35292]: cluster 2024-01-15T22:28:40.777357+0000 mgr.smithi099.cmjeyn (mgr.14184) 49 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:41.946 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:41 smithi162 conmon[39755]: cephadm 2024-01-15T22:28:40.387086+0000 mgr.smithi099.cmjeyn (mgr.14184) 47 : cephadm [INF] Reconfiguring grafana.smithi099 (dependencies changed)... 2024-01-15T22:28:41.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:41 smithi162 conmon[39755]: cephadm 2024-01-15T22:28:40.419076+0000 mgr.smithi099.cmjeyn (mgr.14184) 48 : cephadm 2024-01-15T22:28:41.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:41 smithi162 conmon[39755]: [INF] Reconfiguring daemon grafana.smithi099 on smithi099 2024-01-15T22:28:41.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:41 smithi162 conmon[39755]: cluster 2024-01-15T22:28:40.777357+0000 mgr.smithi099.cmjeyn (mgr.14184) 49 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:42.766 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:28:42.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:42 smithi099 conmon[35292]: audit 2024-01-15T22:28:42.766325+0000 mon.smithi099 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.99:0/779346555' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:28:42.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:42 smithi099 conmon[35292]: cluster 2024-01-15T22:28:42.777746+0000 mgr.smithi099.cmjeyn (mgr.14184) 50 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:43.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:42 smithi162 conmon[39755]: audit 2024-01-15T22:28:42.766325+0000 mon.smithi099 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.99:0/779346555' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:28:43.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:42 smithi162 conmon[39755]: cluster 2024-01-15T22:28:42. 2024-01-15T22:28:43.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:42 smithi162 conmon[39755]: 777746+0000 mgr.smithi099.cmjeyn (mgr.14184) 50 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:43.729 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":9,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:28:44.730 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:28:45.144 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:44 smithi099 conmon[35292]: cluster 2024-01-15T22:28:44.778139+0000 mgr.smithi099.cmjeyn (mgr.14184) 51 : 2024-01-15T22:28:45.144 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:44 smithi099 conmon[35292]: cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:45.196 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:44 smithi162 conmon[39755]: cluster 2024-01-15T22:28:44.778139+0000 mgr.smithi099.cmjeyn (mgr.14184) 51 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:45.196 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:44 smithi162 conmon[39755]: 2024-01-15T22:28:46.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:46 smithi099 conmon[35292]: audit 2024-01-15T22:28:45.331849+0000 mon.smithi099 (mon.0) 250 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:46.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:46 smithi099 conmon[35292]: cephadm 2024-01-15T22:28:45.332493+0000 mgr.smithi099.cmjeyn (mgr.14184) 52 : cephadm [INF] Reconfiguring crash.smithi162 (monmap changed)... 2024-01-15T22:28:46.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:46 smithi099 conmon[35292]: audit 2024-01-15T22:28:45.332761+0000 mon.smithi099 (mon.0) 251 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi162", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-15T22:28:46.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:46 smithi099 conmon[35292]: audit 2024-01-15T22:28:45.333446+0000 mon.smithi099 (mon.0) 252 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:28:46.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:46 smithi099 conmon[35292]: cephadm 2024-01-15T22:28:45.333912+0000 mgr.smithi099.cmjeyn (mgr.14184) 53 : cephadm [INF] Reconfiguring daemon crash.smithi162 on smithi162 2024-01-15T22:28:46.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:46 smithi099 conmon[35292]: audit 2024-01-15T22:28:45.354749+0000 mon.smithi099 (mon.0 2024-01-15T22:28:46.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:46 smithi099 conmon[35292]: ) 253 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:46.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:46 smithi162 conmon[39755]: audit 2024-01-15T22:28:45.331849+0000 mon.smithi099 (mon.0) 250 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:46.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:46 smithi162 conmon[39755]: cephadm 2024-01-15T22:28:45.332493+0000 mgr.smithi099.cmjeyn (mgr.14184) 52 : cephadm [INF] Reconfiguring crash.smithi162 (monmap changed)... 2024-01-15T22:28:46.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:46 smithi162 conmon[39755]: audit 2024-01-15T22:28:45.332761+0000 mon.smithi099 (mon.0) 251 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi162", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-15T22:28:46.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:46 smithi162 conmon[39755]: audit 2024-01-15T22:28:45.333446+0000 mon.smithi099 (mon.0) 252 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:28:46.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:46 smithi162 conmon[39755]: cephadm 2024-01-15T22:28:45.333912+0000 mgr.smithi099.cmjeyn (mgr.14184) 53 : cephadm [INF] Reconfiguring daemon crash.smithi162 on smithi162 2024-01-15T22:28:46.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:46 smithi162 conmon[39755]: audit 2024-01-15T22:28:45.354749+0000 mon.smithi099 (mon.0) 253 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:46.674 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:28:47.185 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":9,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:28:47.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:47 smithi162 conmon[39755]: audit 2024-01-15T22:28:46.601381+0000 mon.smithi099 (mon.0) 254 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:47.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:47 smithi162 conmon[39755]: cephadm 2024-01-15T22:28:46.601965+0000 mgr.smithi099.cmjeyn (mgr.14184) 54 : cephadm [INF] Reconfiguring mgr.smithi162.vbobpt (monmap changed)... 2024-01-15T22:28:47.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:47 smithi162 conmon[39755]: audit 2024-01-15T22:28:46.602255+0000 mon.smithi099 (mon.0) 255 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi162.vbobpt", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-15T22:28:47.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:47 smithi162 conmon[39755]: audit 2024-01-15T22:28:46.602972+0000 mon.smithi099 (mon.0) 256 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-15T22:28:47.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:47 smithi162 conmon[39755]: audit 2024-01-15T22:28:46.603465+0000 mon.smithi099 (mon.0) 257 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:28:47.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:47 smithi162 conmon[39755]: cephadm 2024-01-15T22:28:46.603916+0000 mgr.smithi099.cmjeyn (mgr.14184) 55 : cephadm [INF] Reconfiguring daemon mgr.smithi162.vbobpt on smithi162 2024-01-15T22:28:47.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:47 smithi162 conmon[39755]: audit 2024-01-15T22:28:46.675089+0000 mon.smithi099 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.99:0/998156109' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:28:47.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:47 smithi162 conmon[39755]: cluster 2024-01-15T22:28:46.778550+0000 mgr.smithi099.cmjeyn (mgr.14184) 56 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:47.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:47 smithi099 conmon[35292]: audit 2024-01-15T22:28:46.601381+0000 mon.smithi099 (mon.0) 254 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:47.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:47 smithi099 conmon[35292]: cephadm 2024-01-15T22:28:46.601965+0000 mgr.smithi099.cmjeyn (mgr.14184) 54 : cephadm [INF] Reconfiguring mgr.smithi162.vbobpt (monmap changed)... 2024-01-15T22:28:47.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:47 smithi099 conmon[35292]: audit 2024-01-15T22:28:46.602255+0000 mon.smithi099 (mon.0) 255 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi162.vbobpt", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-15T22:28:47.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:47 smithi099 conmon[35292]: audit 2024-01-15T22:28:46.602972+0000 mon.smithi099 (mon.0) 256 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-15T22:28:47.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:47 smithi099 conmon[35292]: audit 2024-01-15T22:28:46.603465+0000 mon.smithi099 (mon.0) 257 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:28:47.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:47 smithi099 conmon[35292]: cephadm 2024-01-15T22:28:46.603916+0000 mgr.smithi099.cmjeyn (mgr.14184) 55 : cephadm [INF] Reconfiguring daemon mgr.smithi162.vbobpt on smithi162 2024-01-15T22:28:47.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:47 smithi099 conmon[35292]: audit 2024-01-15T22:28:46.675089+0000 mon.smithi099 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.99:0/998156109' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:28:47.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:47 smithi099 conmon[35292]: cluster 2024-01-15T22:28:46.778550+0000 mgr.smithi099.cmjeyn (mgr.14184) 56 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:48.186 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:28:48.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:48 smithi099 conmon[35292]: audit 2024-01-15T22:28:47.797741+0000 mon.smithi099 (mon. 2024-01-15T22:28:48.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:48 smithi099 conmon[35292]: 0) 259 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:48.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:48 smithi099 conmon[35292]: cephadm 2024-01-15T22:28:47.798324+0000 mgr.smithi099.cmjeyn (mgr.14184) 57 : cephadm [INF] Reconfiguring mon.smithi162 (monmap changed)... 2024-01-15T22:28:48.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:48 smithi099 conmon[35292]: audit 2024-01-15T22:28:47.798622+0000 mon.smithi099 (mon.0) 260 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-15T22:28:48.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:48 smithi099 conmon[35292]: audit 2024-01-15T22:28:47.799354+0000 mon.smithi099 (mon.0) 261 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-15T22:28:48.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:48 smithi099 conmon[35292]: audit 2024-01-15T22:28:47.800055+0000 2024-01-15T22:28:48.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:48 smithi099 conmon[35292]: mon.smithi099 (mon.0) 262 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:28:48.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:48 smithi099 conmon[35292]: cephadm 2024-01-15T22:28:48.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:48 smithi099 conmon[35292]: 2024-01-15T22: 2024-01-15T22:28:48.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:48 smithi099 conmon[35292]: 28:47.800688+0000 mgr.smithi099.cmjeyn (mgr.14184 2024-01-15T22:28:48.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:48 smithi099 conmon[35292]: ) 58 : cephadm [INF] Reconfiguring daemon mon.smithi162 on smithi162 2024-01-15T22:28:48.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:48 smithi099 conmon[35292]: cluster 2024-01 2024-01-15T22:28:48.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:48 smithi099 conmon[35292]: -15T22:28:48.779193 2024-01-15T22:28:48.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:48 smithi099 conmon[35292]: +0000 mgr.smithi099.cmjeyn (mgr.14184) 59 : 2024-01-15T22:28:48.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:48 smithi099 conmon[35292]: cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:48.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:48 smithi162 conmon[39755]: audit 2024-01-15T22:28:47.797741+0000 mon.smithi099 (mon.0) 2024-01-15T22:28:48.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:48 smithi162 conmon[39755]: 259 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:48.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:48 smithi162 conmon[39755]: cephadm 2024-01-15T22:28:47.798324+0000 mgr.smithi099.cmjeyn (mgr.14184) 57 : cephadm [INF] Reconfiguring mon.smithi162 (monmap changed)... 2024-01-15T22:28:48.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:48 smithi162 conmon[39755]: audit 2024-01-15T22:28:47.798622+0000 mon.smithi099 (mon 2024-01-15T22:28:48.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:48 smithi162 conmon[39755]: .0) 260 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-15T22:28:48.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:48 smithi162 conmon[39755]: audit 2024-01-15T22:28:47 2024-01-15T22:28:48.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:48 smithi162 conmon[39755]: .799354+0000 mon.smithi099 (mon.0) 261 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-15T22:28:48.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:48 smithi162 conmon[39755]: audit 2024-01-15T 2024-01-15T22:28:48.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:48 smithi162 conmon[39755]: 22:28:47.800055+0000 mon.smithi099 (mon.0) 262 : 2024-01-15T22:28:48.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:48 smithi162 conmon[39755]: audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:28:48.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:48 smithi162 conmon[39755]: cephadm 2024-01-15T22:28:48.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:48 smithi162 conmon[39755]: 2024-01-15T22:28:47.800688+0000 mgr.smithi099.cmjeyn (mgr. 2024-01-15T22:28:48.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:48 smithi162 conmon[39755]: 14184) 58 : cephadm [INF] Reconfiguring daemon mon.smithi162 on smithi162 2024-01-15T22:28:48.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:48 smithi162 conmon[39755]: cluster 2024-01-15T22:28 2024-01-15T22:28:48.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:48 smithi162 conmon[39755]: :48.779193+0000 mgr.smithi099.cmjeyn (mgr.14184) 59 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:48.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:48 smithi162 conmon[39755]: 2024-01-15T22:28:49.975 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:28:50.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:49 smithi099 conmon[35292]: audit 2024-01-15T22:28:48.989838+0000 mon.smithi099 (mon.0) 263 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:50.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:49 smithi099 conmon[35292]: audit 2024-01-15T22:28:48.992604+0000 mon.smithi099 (mon.0) 264 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-15T22:28:50.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:49 smithi099 conmon[35292]: audit 2024-01-15T22:28:48.992969+0000 mgr.smithi099.cmjeyn (mgr.14184) 60 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-15T22:28:50.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:49 smithi099 conmon[35292]: audit 2024-01-15T22:28:48.993700+0000 mon.smithi099 (mon.0) 265 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi099.front.sepia.ceph.com:9093"}]: dispatch 2024-01-15T22:28:50.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:49 smithi099 conmon[35292]: audit 2024-01-15T22:28:48.993942+0000 mgr.smithi099.cmjeyn (mgr.14184) 61 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi099.front.sepia.ceph.com:9093"}]: dispatch 2024-01-15T22:28:50.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:49 smithi099 conmon[35292]: audit 2024-01-15T22:28:48.997211+0000 mon.smithi099 (mon.0) 266 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:50.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:49 smithi099 conmon[35292]: audit 2024-01-15T22:28:49.000823+0000 mon.smithi099 (mon.0) 267 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-15T22:28:50.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:49 smithi099 conmon[35292]: audit 2024-01-15T22:28:49.001130+0000 mgr.smithi099.cmjeyn (mgr.14184) 62 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-15T22:28:50.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:49 smithi099 conmon[35292]: audit 2024-01-15 2024-01-15T22:28:50.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:49 smithi099 conmon[35292]: T22:28:49.004564+0000 mon.smithi099 (mon.0) 268 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi099.front.sepia.ceph.com:3000"}]: dispatch 2024-01-15T22:28:50.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:49 smithi099 conmon[35292]: audit 2024-01-15T22:28:49.004859+0000 mgr.smithi099.cmjeyn (mgr.14184) 63 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi099.front.sepia.ceph.com:3000"}]: dispatch 2024-01-15T22:28:50.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:49 smithi099 conmon[35292]: audit 2024-01-15T22:28:49.010809+0000 mon.smithi099 (mon.0) 269 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:50.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:49 smithi099 conmon[35292]: audit 2024-01-15T22:28:49.015618+0000 mon.smithi099 (mon.0) 270 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-01-15T22:28:50.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:49 smithi099 conmon[35292]: audit 2024-01-15T22:28:49.016153+0000 mgr.smithi099.cmjeyn (mgr.14184) 64 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-01-15T22:28:50.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:49 smithi099 conmon[35292]: audit 2024-01-15T22:28:49.018125+0000 mon.smithi099 (mon.0) 271 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi162.front.sepia.ceph.com:9095"}]: dispatch 2024-01-15T22:28:50.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:49 smithi099 conmon[35292]: audit 2024-01-15T22:28:49.018504+0000 mgr.smithi099.cmjeyn (mgr.14184) 65 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi162.front.sepia.ceph.com:9095"}]: dispatch 2024-01-15T22:28:50.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:49 smithi099 conmon[35292]: audit 2024-01-15T22:28:49.023079+0000 mon.smithi099 (mon.0) 272 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:50.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:49 smithi099 conmon[35292]: audit 2024-01-15T22:28:49.027989+0000 mon.smithi099 (mon.0) 273 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-15T22:28:50.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:49 smithi099 conmon[35292]: audit 2024-01-15T22:28:49.975655+0000 mon.smithi099 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.99:0/2126653429' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:28:50.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: audit 2024-01-15T22:28:48.989838+0000 mon.smithi099 (mon.0) 263 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:50.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: audit 2024-01-15T22:28:48.992604+0000 mon.smithi099 (mon.0) 264 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-15T22:28:50.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: audit 2024-01-15T22:28:48.992969+0000 mgr.smithi099.cmjeyn 2024-01-15T22:28:50.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: (mgr.14184) 60 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-15T22:28:50.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: audit 2024 2024-01-15T22:28:50.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: -01-15T22:28:48.993700+0000 mon.smithi099 (mon.0) 265 2024-01-15T22:28:50.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi099.front.sepia.ceph.com:9093"}]: dispatch 2024-01-15T22:28:50.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: audit 2024-01-15T22:28:48 2024-01-15T22:28:50.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: .993942+0000 mgr.smithi099.cmjeyn (mgr.14184) 61 : audit 2024-01-15T22:28:50.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi099.front.sepia.ceph.com:9093"}]: dispatch 2024-01-15T22:28:50.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: audit 2024-01-15T22:28:50.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: 2024-01-15T22:28:48.997211+0000 mon.smithi099 (mon.0) 2024-01-15T22:28:50.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: 266 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:50.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: audit 2024-01-15T22:28: 2024-01-15T22:28:50.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: 49.000823+0000 mon.smithi099 (mon.0) 267 : audit 2024-01-15T22:28:50.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-15T22:28:50.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: audit 2024-01-15T22:28:49. 2024-01-15T22:28:50.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: 001130+0000 mgr.smithi099.cmjeyn (mgr.14184) 62 : audit 2024-01-15T22:28:50.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-15T22:28:50.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: audit 2024-01-15T22:28:50.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: 2024-01-15T22:28:49.004564+0000 mon.smithi099 (mon 2024-01-15T22:28:50.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: .0) 268 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi099.front.sepia.ceph.com:3000"}]: dispatch 2024-01-15T22:28:50.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: audit 2024-01- 2024-01-15T22:28:50.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: 15T22:28:49.004859+0000 mgr.smithi099.cmjeyn ( 2024-01-15T22:28:50.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: mgr.14184) 63 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi099.front.sepia.ceph.com:3000"}]: dispatch 2024-01-15T22:28:50.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: audit 2024-01-15 2024-01-15T22:28:50.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: T22:28:49.010809+0000 mon.smithi099 (mon. 2024-01-15T22:28:50.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: 0) 269 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:50.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: audit 2024-01-15T22 2024-01-15T22:28:50.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: :28:49.015618+0000 mon.smithi099 (mon.0 2024-01-15T22:28:50.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: ) 270 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-01-15T22:28:50.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: audit 2024-01-15T22:28 2024-01-15T22:28:50.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: :49.016153+0000 mgr.smithi099.cmjeyn (mgr.14184) 64 2024-01-15T22:28:50.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-01-15T22:28:50.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: audit 2024-01-15T22:28:49. 2024-01-15T22:28:50.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: 018125+0000 mon.smithi099 (mon.0) 271 : audit [INF] 2024-01-15T22:28:50.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi162.front.sepia.ceph.com:9095"}]: dispatch 2024-01-15T22:28:50.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: audit 2024 2024-01-15T22:28:50.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: -01-15T22:28:49.018504+0000 mgr.smithi099.cmjeyn (mgr.14184 2024-01-15T22:28:50.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: ) 65 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi162.front.sepia.ceph.com:9095"}]: dispatch 2024-01-15T22:28:50.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: audit 2024-01-15T22: 2024-01-15T22:28:50.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: 28:49.023079+0000 mon.smithi099 (mon.0) 272 2024-01-15T22:28:50.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:50.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: audit 2024-01-15T22:28 2024-01-15T22:28:50.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: :49.027989+0000 mon.smithi099 (mon.0) 273 2024-01-15T22:28:50.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-15T22:28:50.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: audit 2024-01-15 2024-01-15T22:28:50.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: T22:28:49.975655+0000 mon.smithi099 (mon.0 2024-01-15T22:28:50.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:49 smithi162 conmon[39755]: ) 274 : audit [DBG] from='client.? 172.21.15.99:0/2126653429' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:28:50.587 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":9,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:28:51.588 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:28:51.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:51 smithi099 conmon[35292]: audit 2024-01-15T22:28:50.759231+0000 2024-01-15T22:28:51.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:51 smithi099 conmon[35292]: mon.smithi099 (mon.0) 275 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:51.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:51 smithi099 conmon[35292]: cluster 2024-01-15T22:28:50.779557+0000 mgr.smithi099.cmjeyn (mgr.14184) 66 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:52.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:51 smithi162 conmon[39755]: audit 2024-01-15T22:28:50.759231+0000 mon.smithi099 (mon.0) 275 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:52.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:51 smithi162 conmon[39755]: cluster 2024-01-15T22:28:50.779557+0000 mgr.smithi099.cmjeyn (mgr.14184) 66 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:53.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:53 smithi162 conmon[39755]: audit 2024-01-15T22:28:52.008651+0000 mon.smithi099 (mon.0) 276 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:53.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:53 smithi162 conmon[39755]: audit 2024-01-15T22:28:52.375088+0000 mon.smithi099 (mon.0) 277 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:28:53.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:53 smithi162 conmon[39755]: audit 2024-01-15T22:28:52.375765+0000 mon.smithi099 (mon.0) 278 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-15T22:28:53.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:53 smithi162 conmon[39755]: audit 2024-01-15T22:28:52.379736+0000 mon.smithi099 (mon.0) 279 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:53.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:53 smithi162 conmon[39755]: cluster 2024-01-15T22:28:52.380050+0000 mgr.smithi099.cmjeyn (mgr.14184) 67 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:53.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:53 smithi162 conmon[39755]: audit 2024-01-15T22:28:52.383073+0000 mon.smithi099 (mon.0) 280 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-15T22:28:53.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:53 smithi162 conmon[39755]: audit 2024-01-15T22:28:52.385742+0000 mon.smithi099 (mon.0) 281 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-15T22:28:53.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:53 smithi162 conmon[39755]: audit 2024-01-15T22:28:52.386564+0000 mon.smithi099 (mon.0) 282 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-15T22:28:53.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:53 smithi162 conmon[39755]: audit 2024-01-15T22:28:52.386783+0000 mon.smithi099 (mon.0) 2024-01-15T22:28:53.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:53 smithi162 conmon[39755]: 283 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:28:53.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:53 smithi162 conmon[39755]: audit 2024-01-15T22:28:52.387221+0000 mon.smithi099 (mon.0) 284 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:28:53.276 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:53 smithi099 conmon[35292]: audit 2024-01-15T22:28:52.008651+0000 mon.smithi099 (mon.0) 276 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:53.276 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:53 smithi099 conmon[35292]: audit 2024-01-15T22:28:52.375088+0000 mon.smithi099 (mon.0) 277 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:28:53.276 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:53 smithi099 conmon[35292]: audit 2024-01-15T22:28:52.375765+0000 mon.smithi099 (mon.0) 278 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-15T22:28:53.276 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:53 smithi099 conmon[35292]: audit 2024-01-15T22:28:52.379736+0000 mon.smithi099 (mon.0) 279 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:53.276 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:53 smithi099 conmon[35292]: cluster 2024-01-15T22:28:52.380050+0000 mgr.smithi099.cmjeyn (mgr.14184) 67 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:53.276 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:53 smithi099 conmon[35292]: audit 2024-01-15T22:28:52.383073+0000 mon.smithi099 (mon.0) 280 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-15T22:28:53.276 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:53 smithi099 conmon[35292]: audit 2024-01-15T22:28:52.385742+0000 mon.smithi099 (mon.0) 281 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-15T22:28:53.277 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:53 smithi099 conmon[35292]: audit 2024-01-15T22:28:52.386564+0000 mon.smithi099 (mon.0) 282 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-15T22:28:53.277 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:53 smithi099 conmon[35292]: audit 2024-01-15T22:28:52.386783+0000 mon.smithi099 (mon.0) 283 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:28:53.277 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:53 smithi099 conmon[35292]: audit 2024-01-15T22:28:52.387221+0000 mon.smithi099 (mon.0) 284 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:28:53.651 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:28:54.327 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:54 smithi099 conmon[35292]: cluster 2024-01-15T22:28:53.378149+0000 mon.smithi099 (mon.0) 285 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-15T22:28:54.327 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:54 smithi099 conmon[35292]: cluster 2024-01-15T22:28:53.378175+0000 mon.smithi099 (mon.0) 286 : cluster [INF] Cluster is now healthy 2024-01-15T22:28:54.327 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:54 smithi099 conmon[35292]: audit 2024-01-15T22:28:53.652015+0000 mon.smithi162 (mon.1) 4 : audit [DBG] from='client.? 172.21.15.99:0/3394086821' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:28:54.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:54 smithi162 conmon[39755]: cluster 2024-01-15T22:28:53.378149+0000 2024-01-15T22:28:54.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:54 smithi162 conmon[39755]: mon.smithi099 (mon.0) 285 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-15T22:28:54.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:54 smithi162 conmon[39755]: cluster 2024-01-15T22:28:53.378175+0000 mon.smithi099 (mon.0) 286 : cluster [INF] Cluster is now healthy 2024-01-15T22:28:54.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:54 smithi162 conmon[39755]: audit 2024-01-15T22:28:53.652015+0000 mon.smithi162 (mon.1) 4 : audit [DBG] from='client.? 172.21.15.99:0/3394086821' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:28:54.719 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":9,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:28:55.292 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:55 smithi162 conmon[39755]: cluster 2024-01-15T22:28:54.380457+0000 mgr.smithi099.cmjeyn 2024-01-15T22:28:55.293 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:55 smithi162 conmon[39755]: (mgr.14184) 68 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:55.293 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:55 smithi162 conmon[39755]: audit 2024-01-15T22:28:54.903632+0000 mon.smithi162 (mon.1) 5 : audit [INF] from='client.? 172.21.15.162:0/138329409' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4232e749-0be1-4e48-9c10-1a3735d4c554"}]: dispatch 2024-01-15T22:28:55.293 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:55 smithi162 conmon[39755]: audit 2024-01-15T22:28:54.904003+0000 mon.smithi099 (mon.0) 287 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4232e749-0be1-4e48-9c10-1a3735d4c554"}]: dispatch 2024-01-15T22:28:55.293 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:55 smithi162 conmon[39755]: audit 2024-01-15T22:28:54.907562+0000 mon.smithi099 (mon.0) 288 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4232e749-0be1-4e48-9c10-1a3735d4c554"}]': finished 2024-01-15T22:28:55.293 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:55 smithi162 conmon[39755]: cluster 2024-01-15T22:28:54.907663+0000 mon.smithi099 (mon.0) 289 : cluster [DBG] osdmap e10: 1 total, 0 up, 1 in 2024-01-15T22:28:55.293 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:55 smithi162 conmon[39755]: audit 2024-01-15T22:28:54.907763+0000 mon.smithi099 (mon.0) 290 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-15T22:28:55.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:55 smithi099 conmon[35292]: cluster 2024-01-15T22:28:54.380457+0000 mgr.smithi099.cmjeyn (mgr.14184 2024-01-15T22:28:55.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:55 smithi099 conmon[35292]: ) 68 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:55.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:55 smithi099 conmon[35292]: audit 2024-01-15T22:28:54.903632+0000 mon.smithi162 (mon.1) 5 : audit [INF] from='client.? 172.21.15.162:0/138329409' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4232e749-0be1-4e48-9c10-1a3735d4c554"}]: dispatch 2024-01-15T22:28:55.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:55 smithi099 conmon[35292]: audit 2024-01-15T22:28:54.904003+0000 mon.smithi099 (mon.0) 287 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4232e749-0be1-4e48-9c10-1a3735d4c554"}]: dispatch 2024-01-15T22:28:55.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:55 smithi099 conmon[35292]: audit 2024-01-15T22:28:54.907562+0000 mon.smithi099 (mon.0) 288 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4232e749-0be1-4e48-9c10-1a3735d4c554"}]': finished 2024-01-15T22:28:55.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:55 smithi099 conmon[35292]: cluster 2024-01-15T22:28:54.907663+0000 mon.smithi099 (mon.0) 289 : cluster [DBG] osdmap e10: 1 total, 0 up, 1 in 2024-01-15T22:28:55.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:55 smithi099 conmon[35292]: audit 2024-01-15T22:28:54.907763+0000 mon.smithi099 (mon.0) 290 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-15T22:28:55.720 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:28:56.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:56 smithi162 conmon[39755]: audit 2024-01-15T22:28:55.198077+0000 mon.smithi162 (mon.1) 6 : audit [INF] from='client.? 172.21.15.162:0/3884165754' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:28:56.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:56 smithi162 conmon[39755]: audit 2024-01-15T22:28:55.198644+0000 mon.smithi099 (mon.0) 291 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:28:56.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:56 smithi162 conmon[39755]: audit 2024-01-15T22:28:55.202843+0000 mon.smithi099 (mon.0) 292 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-15T22:28:56.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:56 smithi162 conmon[39755]: cluster 2024-01-15T22:28:55.202886+0000 mon.smithi099 (mon.0) 293 : 2024-01-15T22:28:56.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:56 smithi162 conmon[39755]: cluster [DBG] osdmap e11: 0 total, 0 up, 0 in 2024-01-15T22:28:56.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:56 smithi162 conmon[39755]: audit 2024-01-15T22:28:55.898718+0000 2024-01-15T22:28:56.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:56 smithi162 conmon[39755]: mon.smithi099 (mon.0) 294 : audit [INF] from='client.? 172.21.15.99:0/2993915544' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5debeb1c-a955-4138-9d38-631befa5b2b7"}]: dispatch 2024-01-15T22:28:56.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:56 smithi162 conmon[39755]: audit 2024-01-15T22:28:55. 2024-01-15T22:28:56.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:56 smithi162 conmon[39755]: 902562+0000 mon.smithi099 (mon.0) 295 : audit [INF] 2024-01-15T22:28:56.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:56 smithi162 conmon[39755]: from='client.? 172.21.15.99:0/2993915544' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5debeb1c-a955-4138-9d38-631befa5b2b7"}]': finished 2024-01-15T22:28:56.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:56 smithi162 conmon[39755]: cluster 2024-01-15T22:28:56.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:56 smithi162 conmon[39755]: 2024-01-15T22:28:55.902602+0000 mon.smithi099 (mon.0 2024-01-15T22:28:56.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:56 smithi162 conmon[39755]: ) 296 : cluster [DBG] osdmap e12: 1 total, 0 up, 1 in 2024-01-15T22:28:56.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:56 smithi162 conmon[39755]: audit 2024-01-15T22:28 2024-01-15T22:28:56.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:56 smithi162 conmon[39755]: :55.902731+0000 mon.smithi099 (mon.0) 297 : 2024-01-15T22:28:56.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:56 smithi162 conmon[39755]: audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-15T22:28:56.306 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:56 smithi099 conmon[35292]: audit 2024-01-15T22:28:55.198077+0000 mon.smithi162 (mon.1) 6 : audit [INF] from='client.? 172.21.15.162:0/3884165754' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:28:56.306 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:56 smithi099 conmon[35292]: audit 2024-01-15T22:28:55.198644+0000 mon.smithi099 (mon.0) 291 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:28:56.306 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:56 smithi099 conmon[35292]: audit 2024-01-15T22:28:55.202843+0000 mon.smithi099 (mon.0) 292 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-15T22:28:56.306 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:56 smithi099 conmon[35292]: cluster 2024-01-15T22:28:55.202886+0000 mon.smithi099 (mon.0) 293 : cluster [DBG] osdmap e11: 0 total, 0 up, 0 in 2024-01-15T22:28:56.306 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:56 smithi099 conmon[35292]: audit 2024-01-15T22:28:55.898718+0000 mon.smithi099 (mon.0) 294 : audit [INF] from='client.? 172.21.15.99:0/2993915544' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5debeb1c-a955-4138-9d38-631befa5b2b7"}]: dispatch 2024-01-15T22:28:56.306 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:56 smithi099 conmon[35292]: audit 2024-01-15T22:28:55.902562+0000 mon.smithi099 (mon.0) 295 : audit [INF] from='client.? 172.21.15.99:0/2993915544' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5debeb1c-a955-4138-9d38-631befa5b2b7"}]': finished 2024-01-15T22:28:56.306 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:56 smithi099 conmon[35292]: cluster 2024-01-15T22:28:55.902602+0000 mon.smithi099 (mon.0) 296 : cluster [DBG] osdmap e12: 1 total, 0 up, 1 in 2024-01-15T22:28:56.307 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:56 smithi099 conmon[35292]: audit 2024-01-15T22:28:55.902731+0000 mon.smithi099 (mon.0) 297 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-15T22:28:57.351 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: cephadm 2024-01-15T22:28:55.924904+0000 mgr.smithi099.cmjeyn (mgr.14184) 2024-01-15T22:28:57.351 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: 69 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi162', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-15T22:28:57.351 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:28:57.351 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:28:57.351 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: return f(*arg) 2024-01-15T22:28:57.351 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:28:57.351 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:28:57.351 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:28:57.351 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: code, '\n'.join(err))) 2024-01-15T22:28:57.351 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:28:57.351 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp6tw0fido:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpykck1ayr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:28:57.351 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:28:57.352 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:28:57.352 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:28:57.352 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4232e749-0be1-4e48-9c10-1a3735d4c554 2024-01-15T22:28:57.352 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:28:57.352 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:28:57.352 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:28:57.352 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:28:57.352 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:28:57.352 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr self.prepare() 2024-01-15T22:28:57.353 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:57.353 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:57.353 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:28:57.353 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:28:57.353 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:28:57.353 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:28:57.353 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:28:57.353 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:28:57.353 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:28:57.353 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:28:57.353 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:28:57.353 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:28:57.354 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:28:57.354 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:28:57.354 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:28:57.354 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:28:57.354 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:28:57.354 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:28:57.354 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:28:57.354 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:28:57.354 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:28:57.354 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:28:57.354 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:28:57.354 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:28:57.354 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:28:57.354 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:28:57.354 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:28:57.354 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:57.355 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:57.355 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:28:57.355 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:28:57.355 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:28:57.355 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:28:57.355 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:57.355 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:57.355 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:28:57.355 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:28:57.355 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:28:57.355 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:28:57.355 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:28:57.355 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:28:57.355 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:28:57.355 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:28:57.355 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:57.356 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:57.356 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:28:57.356 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:28:57.356 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:28:57.356 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:28:57.356 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:28:57.356 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:28:57.356 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:28:57.356 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: main() 2024-01-15T22:28:57.356 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:28:57.356 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: r = ctx.func(ctx) 2024-01-15T22:28:57.356 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:28:57.356 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:28:57.356 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:28:57.356 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:28:57.356 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:28:57.357 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:28:57.357 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda 2024-01-15T22:28:57.357 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: 83f90d2", line 1927, in _validate_fsid 2024-01-15T22:28:57.357 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:28:57.357 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:28:57.357 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:28:57.357 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:28:57.357 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:28:57.357 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp6tw0fido:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpykck1ayr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:28:57.357 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: audit 2024-01-15T22:28:56.211936+0000 mon.smithi099 (mon.0) 298 : audit [INF] from='client.? 172.21.15.99:0/161332521' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:28:57.357 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: audit 2024-01-15T22:28:56.215412+0000 mon.smithi099 (mon.0) 299 : audit [INF] from='client.? 172.21.15.99:0/161332521' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-15T22:28:57.357 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: cluster 2024-01-15T22:28:56.215484+0000 mon.smithi099 (mon.0) 300 : cluster [DBG] osdmap e13: 0 total, 0 up, 0 in 2024-01-15T22:28:57.357 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:57 smithi099 conmon[35292]: cluster 2024-01-15T22:28:56.380730+0000 mgr.smithi099.cmjeyn (mgr.14184) 70 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:57.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: cephadm 2024-01-15T22:28:55.924904+0000 mgr.smithi099.cmjeyn (mgr.14184) 69 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi162', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-15T22:28:57.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:28:57.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:28:57.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: return f(*arg) 2024-01-15T22:28:57.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:28:57.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:28:57.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:28:57.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: code, '\n'.join(err))) 2024-01-15T22:28:57.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:28:57.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp6tw0fido:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpykck1ayr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:28:57.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:28:57.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:28:57.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:28:57.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4232e749-0be1-4e48-9c10-1a3735d4c554 2024-01-15T22:28:57.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:28:57.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:28:57.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:28:57.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:28:57.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:28:57.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr self.prepare() 2024-01-15T22:28:57.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:57.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:57.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:28:57.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:28:57.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:28:57.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:28:57.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:28:57.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:28:57.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:28:57.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:28:57.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:28:57.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:28:57.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:28:57.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:28:57.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:28:57.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:28:57.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:28:57.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:28:57.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:28:57.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:28:57.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:28:57.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:28:57.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:28:57.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:28:57.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:28:57.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:28:57.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:28:57.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:57.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:57.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:28:57.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:28:57.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:28:57.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:28:57.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:57.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:57.451 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:28:57.451 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:28:57.451 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:28:57.451 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:28:57.451 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:28:57.451 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:28:57.451 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:28:57.451 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:28:57.451 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:57.451 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:57.451 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:28:57.452 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:28:57.452 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:28:57.452 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:28:57.452 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:28:57.452 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:28:57.452 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:28:57.452 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: main() 2024-01-15T22:28:57.452 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:28:57.452 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: r = ctx.func(ctx) 2024-01-15T22:28:57.453 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:28:57.453 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:28:57.453 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:28:57.453 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:28:57.453 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:28:57.453 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:28:57.453 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774 2024-01-15T22:28:57.453 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:28:57.453 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:28:57.453 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:28:57.453 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:28:57.453 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:28:57.453 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:28:57.454 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp6tw0fido:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpykck1ayr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:28:57.454 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: audit 2024-01-15T22:28:56.211936+0000 mon.smithi099 (mon.0) 298 : audit [INF] from='client.? 172.21.15.99:0/161332521' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:28:57.454 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: audit 2024-01-15T22:28:56.215412+0000 mon.smithi099 (mon.0) 299 : audit [INF] from='client.? 172.21.15.99:0/161332521' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-15T22:28:57.454 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: cluster 2024-01-15T22:28:56.215484+0000 mon.smithi099 (mon.0) 300 : cluster [DBG] osdmap e13: 0 total, 0 up, 0 in 2024-01-15T22:28:57.454 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:57 smithi162 conmon[39755]: cluster 2024-01-15T22:28:56.380730+0000 mgr.smithi099.cmjeyn (mgr.14184) 70 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:58.153 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:28:58.450 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: cephadm 2024-01-15T22:28:57.378349+0000 mgr.smithi099.cmjeyn (mgr.14184) 71 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi162', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-15T22:28:58.450 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:28:58.450 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:28:58.450 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: return f(*arg) 2024-01-15T22:28:58.450 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:28:58.450 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:28:58.450 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:28:58.450 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: code, '\n'.join(err))) 2024-01-15T22:28:58.450 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi099/config 2024-01-15T22:28:58.451 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpqn3imsum:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpzs49xant:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:28:58.451 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:28:58.451 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:28:58.451 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:28:58.451 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 5debeb1c-a955-4138-9d38-631befa5b2b7 2024-01-15T22:28:58.451 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:28:58.451 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:28:58.451 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:28:58.451 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:28:58.451 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:28:58.451 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr self.prepare() 2024-01-15T22:28:58.451 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:58.451 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:58.451 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:28:58.451 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:28:58.452 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:28:58.452 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:28:58.452 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:28:58.452 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:28:58.452 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:28:58.452 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:28:58.452 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:28:58.452 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:28:58.452 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:28:58.452 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:28:58.452 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:28:58.452 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:28:58.452 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:28:58.452 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:28:58.453 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:28:58.453 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:28:58.453 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:28:58.453 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:28:58.453 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:28:58.453 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:28:58.453 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:28:58.453 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:28:58.453 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:28:58.453 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:58.453 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:58.453 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:28:58.453 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:28:58.453 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:28:58.453 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:28:58.454 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:58.454 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:58.454 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:28:58.454 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:28:58.454 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:28:58.454 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:28:58.454 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:28:58.454 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:28:58.454 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:28:58.454 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:28:58.454 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:58.454 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:58.454 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:28:58.454 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:28:58.454 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:28:58.455 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:28:58.455 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:28:58.455 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:28:58.455 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:28:58.455 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: main() 2024-01-15T22:28:58.455 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:28:58.455 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: r = ctx.func(ctx) 2024-01-15T22:28:58.455 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:28:58.455 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:28:58.455 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:28:58.455 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:28:58.455 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:28:58.455 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:28:58.455 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774 2024-01-15T22:28:58.456 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:28:58.456 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:28:58.456 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:28:58.456 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:28:58.456 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:28:58.456 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:28:58.456 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpqn3imsum:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpzs49xant:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:28:58.456 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: cephadm 2024-01-15T22:28:57.380319+0000 mgr.smithi099.cmjeyn (mgr.14184) 72 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-15T22:28:58.456 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: service_id: all-available-devices 2024-01-15T22:28:58.456 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: service_name: osd.all-available-devices 2024-01-15T22:28:58.456 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: placement: 2024-01-15T22:28:58.456 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: host_pattern: '*' 2024-01-15T22:28:58.457 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: spec: 2024-01-15T22:28:58.457 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: data_devices: 2024-01-15T22:28:58.457 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: all: true 2024-01-15T22:28:58.457 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: filter_logic: AND 2024-01-15T22:28:58.457 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: objectstore: bluestore 2024-01-15T22:28:58.457 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:28:58.457 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp6tw0fido:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpykck1ayr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:28:58.457 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:28:58.457 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:28:58.457 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:28:58.457 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4232e749-0be1-4e48-9c10-1a3735d4c554 2024-01-15T22:28:58.457 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:28:58.457 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:28:58.457 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:28:58.458 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:28:58.458 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:28:58.458 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr self.prepare() 2024-01-15T22:28:58.458 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:58.458 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:58.458 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:28:58.458 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:28:58.458 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:28:58.458 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:28:58.458 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:28:58.458 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:28:58.458 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:28:58.458 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:28:58.458 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:28:58.458 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:28:58.458 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:28:58.459 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:28:58.459 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:28:58.459 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:28:58.459 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:28:58.459 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:28:58.459 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:28:58.459 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:28:58.459 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:28:58.459 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:28:58.459 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:28:58.459 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:28:58.459 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:28:58.459 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:28:58.459 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:28:58.460 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:58.461 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:58.461 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:28:58.461 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:28:58.461 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:28:58.461 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:28:58.461 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:58.461 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:58.461 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:28:58.461 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:28:58.461 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:28:58.461 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:28:58.461 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:28:58.461 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:28:58.461 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:28:58.461 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:28:58.462 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:58.462 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:58.462 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lv 2024-01-15T22:28:58.462 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: m/zap.py", line 302, in zap_osd 2024-01-15T22:28:58.462 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:28:58.462 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:28:58.462 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:28:58.462 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:28:58.462 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:28:58.462 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:28:58.462 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: main() 2024-01-15T22:28:58.462 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:28:58.462 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: r = ctx.func(ctx) 2024-01-15T22:28:58.462 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:28:58.462 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:28:58.463 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:28:58.463 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:28:58.463 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:28:58.463 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:28:58.463 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:28:58.463 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:28:58.463 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:28:58.463 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:28:58.463 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:28:58.463 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:28:58.463 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp6tw0fido:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpykck1ayr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:28:58.463 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:28:58.463 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-15T22:28:58.463 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: if self._apply_service(spec): 2024-01-15T22:28:58.463 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-15T22:28:58.463 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-15T22:28:58.463 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-15T22:28:58.464 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-15T22:28:58.464 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-15T22:28:58.464 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-15T22:28:58.464 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-15T22:28:58.464 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-15T22:28:58.464 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-15T22:28:58.464 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: raise self._value 2024-01-15T22:28:58.464 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-15T22:28:58.464 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: result = (True, func(*args, **kwds)) 2024-01-15T22:28:58.464 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-15T22:28:58.464 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: return list(map(*args)) 2024-01-15T22:28:58.464 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:28:58.464 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: return f(*arg) 2024-01-15T22:28:58.465 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:28:58.465 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:28:58.465 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:28:58.465 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: code, '\n'.join(err))) 2024-01-15T22:28:58.465 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:28:58.465 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp6tw0fido:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpykck1ayr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:28:58.465 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:28:58.465 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:28:58.465 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:28:58.465 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4232e749-0be1-4e48-9c10-1a3735d4c554 2024-01-15T22:28:58.465 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:28:58.465 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:28:58.465 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:28:58.465 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:28:58.465 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:28:58.465 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr self.prepare() 2024-01-15T22:28:58.466 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:58.466 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:58.466 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:28:58.466 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:28:58.466 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:28:58.466 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:28:58.466 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:28:58.466 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:28:58.466 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:28:58.466 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:28:58.466 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:28:58.466 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:28:58.466 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:28:58.466 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:28:58.466 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:28:58.466 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:28:58.466 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:28:58.467 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr F 2024-01-15T22:28:58.467 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: ile "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:28:58.467 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:28:58.467 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:28:58.467 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:28:58.467 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:28:58.467 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:28:58.467 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:28:58.467 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:28:58.467 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:28:58.467 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:28:58.467 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:58.467 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:58.467 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:28:58.467 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:28:58.467 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:28:58.467 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:28:58.468 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:58.468 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:58.468 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:28:58.468 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:28:58.468 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:28:58.468 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:28:58.468 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:28:58.468 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:28:58.468 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:28:58.468 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:28:58.468 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:58.468 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:58.468 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:28:58.468 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:28:58.468 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:28:58.468 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:28:58.469 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:28:58.469 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:28:58.469 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:28:58.469 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: main() 2024-01-15T22:28:58.469 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:28:58.469 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: r = ctx.func(ctx) 2024-01-15T22:28:58.469 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:28:58.469 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:28:58.469 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:28:58.469 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:28:58.469 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:28:58.469 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:28:58.469 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:28:58.469 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:28:58.469 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:28:58.469 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:28:58.470 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:28:58.470 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:28:58.470 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp6tw0fido:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpykck1ayr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:28:58.470 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: cluster 2024-01-15T22:28:57.381027+0000 mgr.smithi099.cmjeyn (mgr.14184) 73 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:58.470 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: audit 2024-01-15T22:28:57.382188+0000 mon.smithi099 (mon.0) 301 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-15T22:28:58.470 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:58 smithi099 conmon[35292]: audit 2024-01-15T22:28:58.153865+0000 mon.smithi099 (mon.0) 302 : audit [DBG] from='client.? 172.21.15.99:0/1715023976' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:28:58.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: cephadm 2024-01-15T22:28:57.378349+0000 mgr.smithi099.cmjeyn (mgr.14184) 71 : 2024-01-15T22:28:58.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi162', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-15T22:28:58.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:28:58.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:28:58.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: return f(*arg) 2024-01-15T22:28:58.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:28:58.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:28:58.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:28:58.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: code, '\n'.join(err))) 2024-01-15T22:28:58.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi099/config 2024-01-15T22:28:58.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpqn3imsum:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpzs49xant:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:28:58.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:28:58.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:28:58.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:28:58.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 5debeb1c-a955-4138-9d38-631befa5b2b7 2024-01-15T22:28:58.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:28:58.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:28:58.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:28:58.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:28:58.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:28:58.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr self.prepare() 2024-01-15T22:28:58.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:58.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:58.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:28:58.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:28:58.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:28:58.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:28:58.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:28:58.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:28:58.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:28:58.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:28:58.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:28:58.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:28:58.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:28:58.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:28:58.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:28:58.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:28:58.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:28:58.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:28:58.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:28:58.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:28:58.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:28:58.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:28:58.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:28:58.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:28:58.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:28:58.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:28:58.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:28:58.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:58.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:58.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:28:58.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:28:58.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:28:58.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:28:58.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:58.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:58.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:28:58.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:28:58.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:28:58.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:28:58.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:28:58.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:28:58.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:28:58.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:28:58.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:58.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:58.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:28:58.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:28:58.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:28:58.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:28:58.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:28:58.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:28:58.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:28:58.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: main() 2024-01-15T22:28:58.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:28:58.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: r = ctx.func(ctx) 2024-01-15T22:28:58.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:28:58.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:28:58.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:28:58.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:28:58.704 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:28:58.704 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:28:58.704 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90 2024-01-15T22:28:58.704 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: d2", line 1927, in _validate_fsid 2024-01-15T22:28:58.704 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:28:58.704 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:28:58.704 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:28:58.704 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:28:58.704 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:28:58.704 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpqn3imsum:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpzs49xant:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:28:58.704 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: cephadm 2024-01-15T22:28:57.380319+0000 mgr.smithi099.cmjeyn (mgr.14184) 72 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-15T22:28:58.704 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: service_id: all-available-devices 2024-01-15T22:28:58.704 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: service_name: osd.all-available-devices 2024-01-15T22:28:58.704 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: placement: 2024-01-15T22:28:58.705 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: host_pattern: '*' 2024-01-15T22:28:58.705 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: spec: 2024-01-15T22:28:58.705 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: data_devices: 2024-01-15T22:28:58.705 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: all: true 2024-01-15T22:28:58.705 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: filter_logic: AND 2024-01-15T22:28:58.705 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: objectstore: bluestore 2024-01-15T22:28:58.705 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:28:58.705 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp6tw0fido:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpykck1ayr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:28:58.705 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:28:58.705 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:28:58.705 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:28:58.705 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4232e749-0be1-4e48-9c10-1a3735d4c554 2024-01-15T22:28:58.705 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:28:58.705 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:28:58.706 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:28:58.706 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:28:58.706 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:28:58.706 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr self.prepare() 2024-01-15T22:28:58.706 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:58.706 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:58.706 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:28:58.706 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:28:58.706 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:28:58.706 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:28:58.706 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:28:58.706 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:28:58.706 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:28:58.706 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:28:58.706 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:28:58.707 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:28:58.707 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:28:58.707 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:28:58.707 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:28:58.707 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:28:58.707 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:28:58.707 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:28:58.707 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:28:58.707 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:28:58.707 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:28:58.707 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:28:58.707 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:28:58.707 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:28:58.707 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:28:58.707 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:28:58.707 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:28:58.708 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:58.708 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:58.708 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:28:58.708 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:28:58.708 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:28:58.708 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:28:58.708 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:58.708 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:58.708 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:28:58.708 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:28:58.708 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:28:58.708 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:28:58.708 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:28:58.708 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:28:58.708 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:28:58.709 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:28:58.709 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:58.709 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:58.709 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:28:58.709 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr devices = find_associa 2024-01-15T22:28:58.709 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: ted_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:28:58.709 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:28:58.709 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:28:58.709 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:28:58.709 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:28:58.709 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:28:58.709 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: main() 2024-01-15T22:28:58.709 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:28:58.709 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: r = ctx.func(ctx) 2024-01-15T22:28:58.709 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:28:58.710 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:28:58.710 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:28:58.710 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:28:58.710 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:28:58.710 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:28:58.710 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:28:58.710 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:28:58.710 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:28:58.710 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:28:58.710 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:28:58.710 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:28:58.710 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp6tw0fido:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpykck1ayr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:28:58.710 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:28:58.710 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-15T22:28:58.711 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: if self._apply_service(spec): 2024-01-15T22:28:58.711 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-15T22:28:58.711 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-15T22:28:58.711 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-15T22:28:58.711 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-15T22:28:58.711 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-15T22:28:58.711 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-15T22:28:58.711 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-15T22:28:58.711 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-15T22:28:58.711 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-15T22:28:58.711 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: raise self._value 2024-01-15T22:28:58.711 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-15T22:28:58.711 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: result = (True, func(*args, **kwds)) 2024-01-15T22:28:58.711 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-15T22:28:58.712 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: return list(map(*args)) 2024-01-15T22:28:58.712 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:28:58.712 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: return f(*arg) 2024-01-15T22:28:58.712 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:28:58.712 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:28:58.712 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:28:58.713 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: code, '\n'.join(err))) 2024-01-15T22:28:58.713 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:28:58.713 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp6tw0fido:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpykck1ayr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:28:58.713 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:28:58.713 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:28:58.713 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:28:58.713 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4232e749-0be1-4e48-9c10-1a3735d4c554 2024-01-15T22:28:58.713 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:28:58.713 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:28:58.713 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:28:58.713 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:28:58.713 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:28:58.713 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr self.prepare() 2024-01-15T22:28:58.713 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:58.714 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:58.714 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:28:58.714 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:28:58.714 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:28:58.714 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:28:58.714 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:28:58.714 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:28:58.714 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:28:58.714 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:28:58.714 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:28:58.714 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:28:58.714 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:28:58.714 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:28:58.714 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:28:58.714 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:28:58.715 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:28:58.715 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in 2024-01-15T22:28:58.715 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: newfunc 2024-01-15T22:28:58.715 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:28:58.715 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:28:58.715 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:28:58.715 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:28:58.715 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:28:58.715 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:28:58.715 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:28:58.715 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:28:58.715 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:28:58.715 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:58.715 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:58.715 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:28:58.715 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:28:58.716 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:28:58.716 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:28:58.716 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:58.716 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:58.716 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:28:58.716 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:28:58.716 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:28:58.716 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:28:58.716 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:28:58.716 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:28:58.716 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:28:58.716 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:28:58.716 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:28:58.716 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:28:58.716 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:28:58.717 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:28:58.717 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:28:58.717 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:28:58.717 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:28:58.717 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:28:58.717 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:28:58.717 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: main() 2024-01-15T22:28:58.717 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:28:58.717 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: r = ctx.func(ctx) 2024-01-15T22:28:58.717 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:28:58.717 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:28:58.717 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:28:58.717 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:28:58.717 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:28:58.717 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:28:58.718 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:28:58.718 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:28:58.718 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:28:58.718 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:28:58.718 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:28:58.718 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:28:58.718 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp6tw0fido:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpykck1ayr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:28:58.718 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: cluster 2024-01-15T22:28:57.381027+0000 mgr.smithi099.cmjeyn (mgr.14184) 73 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:28:58.718 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: audit 2024-01-15T22:28:57.382188+0000 mon.smithi099 (mon.0) 301 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-15T22:28:58.718 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:58 smithi162 conmon[39755]: audit 2024-01-15T22:28:58.153865+0000 mon.smithi099 (mon.0) 302 : audit [DBG] from='client.? 172.21.15.99:0/1715023976' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:28:59.039 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:28:59.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:59 smithi099 conmon[35292]: cluster 2024-01-15T22:28:58.215378+0000 mon.smithi099 (mon.0) 2024-01-15T22:28:59.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:59 smithi099 conmon[35292]: 303 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-15T22:28:59.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:28:59 smithi099 conmon[35292]: audit 2024-01-15T22:28:59.212440+0000 mon.smithi099 (mon.0) 304 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:28:59.692 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:59 smithi162 conmon[39755]: cluster 2024-01-15T22:28:58.215378+0000 mon.smithi099 (mon.0) 303 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-15T22:28:59.693 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:28:59 smithi162 conmon[39755]: audit 2024-01-15T22:28:59.212440+0000 mon.smithi099 (mon.0) 304 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:29:00.040 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:29:00.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:00 smithi099 conmon[35292]: cluster 2024-01-15T22:28:59.381450+0000 mgr.smithi099.cmjeyn (mgr.14184) 74 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:00.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:00 smithi162 conmon[39755]: cluster 2024-01-15T22:28:59.381450+0000 mgr.smithi099.cmjeyn (mgr.14184) 74 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:01.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:01 smithi162 conmon[39755]: audit 2024-01-15T22:29:00.487295+0000 mon.smithi099 (mon.0) 305 2024-01-15T22:29:01.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:01 smithi162 conmon[39755]: : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:29:01.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:01 smithi162 conmon[39755]: audit 2024-01-15T22:29:00.487951+0000 mon.smithi099 (mon.0) 306 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:29:01.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:01 smithi162 conmon[39755]: audit 2024-01-15T22:29:00.488482+0000 mon.smithi099 (mon.0) 307 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-15T22:29:01.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:01 smithi162 conmon[39755]: audit 2024-01-15T22:29:00.492018+0000 mon.smithi099 (mon.0) 308 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:29:01.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:01 smithi162 conmon[39755]: cluster 2024-01-15T22:29:00.492289+0000 mgr.smithi099.cmjeyn (mgr.14184) 75 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:01.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:01 smithi162 conmon[39755]: audit 2024-01-15T22:29:00.495291+0000 mon.smithi099 (mon. 2024-01-15T22:29:01.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:01 smithi162 conmon[39755]: 0) 309 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-15T22:29:01.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:01 smithi162 conmon[39755]: audit 2024-01- 2024-01-15T22:29:01.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:01 smithi162 conmon[39755]: 15T22:29:00.497750+0000 mon.smithi099 (mon 2024-01-15T22:29:01.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:01 smithi162 conmon[39755]: .0) 310 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-15T22:29:01.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:01 smithi162 conmon[39755]: 2024-01-15T22:29:01.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:01 smithi162 conmon[39755]: audit 2024-01- 2024-01-15T22:29:01.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:01 smithi162 conmon[39755]: 15T22:29:00.498729+0000 2024-01-15T22:29:01.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:01 smithi162 conmon[39755]: mon.smithi099 (mon.0) 311 : audit 2024-01-15T22:29:01.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:01 smithi162 conmon[39755]: [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-15T22:29:01.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:01 smithi162 conmon[39755]: audit 2024-01-15T22:29:01.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:01 smithi162 conmon[39755]: 2024-01- 2024-01-15T22:29:01.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:01 smithi162 conmon[39755]: 15T22:29:00.498960+0000 2024-01-15T22:29:01.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:01 smithi162 conmon[39755]: mon.smithi099 (mon.0) 312 : 2024-01-15T22:29:01.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:01 smithi162 conmon[39755]: audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:29:01.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:01 smithi162 conmon[39755]: audit 2024-01-15T22:29:01.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:01 smithi162 conmon[39755]: 2024-01-15T22 2024-01-15T22:29:01.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:01 smithi162 conmon[39755]: :29:00.499373+0000 mon.smithi099 (mon.0) 2024-01-15T22:29:01.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:01 smithi162 conmon[39755]: 313 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:29:01.780 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:01 smithi099 conmon[35292]: audit 2024-01-15T22:29:00.487295+0000 mon.smithi099 (mon.0) 305 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:29:01.781 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:01 smithi099 conmon[35292]: audit 2024-01-15T22:29:00.487951+0000 mon.smithi099 (mon.0) 306 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:29:01.781 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:01 smithi099 conmon[35292]: audit 2024-01-15T22:29:00.488482+0000 mon.smithi099 (mon.0) 307 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-15T22:29:01.781 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:01 smithi099 conmon[35292]: audit 2024-01-15T22:29:00.492018+0000 mon.smithi099 (mon.0) 308 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:29:01.781 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:01 smithi099 conmon[35292]: cluster 2024-01-15T22:29:00.492289+0000 mgr.smithi099.cmjeyn (mgr.14184) 75 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:01.781 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:01 smithi099 conmon[35292]: audit 2024-01-15T22:29:00.495291+0000 mon.smithi099 (mon.0) 309 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-15T22:29:01.781 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:01 smithi099 conmon[35292]: audit 2024-01-15T22:29:00.497750+0000 mon.smithi099 (mon.0) 310 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-15T22:29:01.781 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:01 smithi099 conmon[35292]: audit 2024-01-15T22:29:00.498729+0000 mon.smithi099 (mon.0) 311 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-15T22:29:01.781 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:01 smithi099 conmon[35292]: audit 2024-01-15T22:29:00.498960+0000 mon.smithi099 (mon.0) 312 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:29:01.781 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:01 smithi099 conmon[35292]: audit 2024-01-15T22:29:00.499373+0000 mon.smithi099 (mon.0) 313 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:29:02.148 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:29:02.838 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:02 smithi099 conmon[35292]: cluster 2024-01-15T22:29:01.490958+0000 mon.smithi099 (mon.0) 2024-01-15T22:29:02.838 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:02 smithi099 conmon[35292]: 314 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-15T22:29:02.838 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:02 smithi099 conmon[35292]: cluster 2024-01-15T22:29:01.490985+0000 mon.smithi099 (mon.0) 315 : cluster [INF] Cluster is now healthy 2024-01-15T22:29:02.838 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:02 smithi099 conmon[35292]: audit 2024-01-15T22:29:02.148654+0000 mon.smithi099 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.99:0/566568501' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:29:02.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:02 smithi162 conmon[39755]: cluster 2024-01-15T22:29:01.490958+0000 mon.smithi099 (mon.0) 314 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-15T22:29:02.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:02 smithi162 conmon[39755]: cluster 2024-01-15T22:29:01.490985+0000 mon.smithi099 (mon.0) 315 : cluster [INF] Cluster is now healthy 2024-01-15T22:29:02.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:02 smithi162 conmon[39755]: audit 2024-01-15T22:29:02.148654+0000 mon.smithi099 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.99:0/566568501' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:29:03.073 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:29:03.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:03 smithi099 conmon[35292]: cluster 2024-01-15T22:29:02.492471+0000 mgr.smithi099.cmjeyn (mgr.14184) 76 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:03.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:03 smithi099 conmon[35292]: audit 2024-01-15T22:29:03.159811+0000 mon.smithi162 (mon.1) 7 : audit [INF] from='client.? 172.21.15.162:0/1759962062' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "339f1c8c-0c2c-46f4-bd37-a13d21334cb9"}]: dispatch 2024-01-15T22:29:03.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:03 smithi099 conmon[35292]: audit 2024-01-15T22:29:03.160168+0000 mon.smithi099 (mon.0) 317 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "339f1c8c-0c2c-46f4-bd37-a13d21334cb9"}]: dispatch 2024-01-15T22:29:03.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:03 smithi099 conmon[35292]: audit 2024-01-15T22:29:03.166346+0000 mon.smithi099 (mon.0) 318 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "339f1c8c-0c2c-46f4-bd37-a13d21334cb9"}]': finished 2024-01-15T22:29:03.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:03 smithi099 conmon[35292]: cluster 2024-01-15T22:29:03.166389+0000 mon.smithi099 (mon.0) 319 : cluster [DBG] osdmap e14: 1 total, 0 up, 1 in 2024-01-15T22:29:03.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:03 smithi099 conmon[35292]: audit 2024-01-15T22:29:03.166466+0000 mon.smithi099 (mon.0) 320 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-15T22:29:03.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:03 smithi099 conmon[35292]: audit 2024-01-15T22:29:03.452315+0000 mon.smithi162 (mon.1) 8 : audit [INF] from='client.? 172.21.15.162:0/2403713555' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:29:03.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:03 smithi099 conmon[35292]: audit 2024-01-15T22:29:03.452745+0000 mon.smithi099 (mon.0) 321 : audit 2024-01-15T22:29:03.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:03 smithi099 conmon[35292]: [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:29:03.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:03 smithi099 conmon[35292]: audit 2024-01-15T22:29:03.460893+0000 mon.smithi099 (mon.0) 322 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-15T22:29:03.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:03 smithi099 conmon[35292]: cluster 2024-01-15T22:29:03.460941+0000 mon.smithi099 (mon.0) 323 : cluster [DBG] osdmap e15: 0 total, 0 up, 0 in 2024-01-15T22:29:03.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:03 smithi162 conmon[39755]: cluster 2024-01-15T22:29:02.492471+0000 mgr.smithi099.cmjeyn (mgr.14184) 2024-01-15T22:29:03.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:03 smithi162 conmon[39755]: 76 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:03.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:03 smithi162 conmon[39755]: audit 2024-01-15T22:29:03.159811+0000 mon.smithi162 (mon.1) 7 : audit [INF] from='client.? 172.21.15.162:0/1759962062' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "339f1c8c-0c2c-46f4-bd37-a13d21334cb9"}]: dispatch 2024-01-15T22:29:03.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:03 smithi162 conmon[39755]: audit 2024-01-15T22:29:03.160168+0000 mon.smithi099 (mon.0) 317 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "339f1c8c-0c2c-46f4-bd37-a13d21334cb9"}]: dispatch 2024-01-15T22:29:03.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:03 smithi162 conmon[39755]: audit 2024-01-15T22:29:03.166346+0000 mon.smithi099 (mon.0) 318 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "339f1c8c-0c2c-46f4-bd37-a13d21334cb9"}]': finished 2024-01-15T22:29:03.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:03 smithi162 conmon[39755]: cluster 2024-01-15T22:29:03.166389+0000 mon.smithi099 (mon.0) 319 : cluster [DBG] osdmap e14: 1 total, 0 up, 1 in 2024-01-15T22:29:03.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:03 smithi162 conmon[39755]: audit 2024-01 2024-01-15T22:29:03.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:03 smithi162 conmon[39755]: -15T22:29:03.166466+0000 mon.smithi099 (mon 2024-01-15T22:29:03.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:03 smithi162 conmon[39755]: .0) 320 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-15T22:29:03.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:03 smithi162 conmon[39755]: audit 2024-01-15T22: 2024-01-15T22:29:03.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:03 smithi162 conmon[39755]: 29:03.452315+0000 mon.smithi162 (mon.1) 8 2024-01-15T22:29:03.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:03 smithi162 conmon[39755]: : audit [INF] from='client.? 172.21.15.162:0/2403713555' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:29:03.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:03 smithi162 conmon[39755]: audit 2024-01-15T22:29: 2024-01-15T22:29:03.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:03 smithi162 conmon[39755]: 03.452745+0000 mon.smithi099 (mon.0) 321 : audit 2024-01-15T22:29:03.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:03 smithi162 conmon[39755]: [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:29:03.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:03 smithi162 conmon[39755]: audit 2024-01-15T22:29:03. 2024-01-15T22:29:03.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:03 smithi162 conmon[39755]: 460893+0000 mon.smithi099 (mon.0) 322 : audit [INF] 2024-01-15T22:29:03.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:03 smithi162 conmon[39755]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-15T22:29:03.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:03 smithi162 conmon[39755]: cluster 2024-01-15T22:29:03.460941 2024-01-15T22:29:03.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:03 smithi162 conmon[39755]: +0000 mon.smithi099 (mon.0) 323 : cluster [DBG] 2024-01-15T22:29:03.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:03 smithi162 conmon[39755]: osdmap e15: 0 total, 0 up, 0 in 2024-01-15T22:29:04.073 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:29:04.754 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:04 smithi099 conmon[35292]: audit 2024-01-15T22:29:04.292947+0000 mon.smithi099 2024-01-15T22:29:04.754 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:04 smithi099 conmon[35292]: (mon.0) 324 : audit [INF] from='client.? 172.21.15.99:0/1447421086' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c768cbd1-4c1d-4ba9-8273-a4b144d392ae"}]: dispatch 2024-01-15T22:29:04.754 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:04 smithi099 conmon[35292]: audit 2024-01-15T22:29:04.296568+0000 mon.smithi099 (mon.0) 325 : audit [INF] from='client.? 172.21.15.99:0/1447421086' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c768cbd1-4c1d-4ba9-8273-a4b144d392ae"}]': finished 2024-01-15T22:29:04.754 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:04 smithi099 conmon[35292]: cluster 2024-01-15T22:29:04.296611+0000 mon.smithi099 (mon.0) 326 : cluster [DBG] osdmap e16: 1 total, 0 up, 1 in 2024-01-15T22:29:04.754 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:04 smithi099 conmon[35292]: audit 2024-01-15T22:29:04.296725+0000 mon.smithi099 (mon.0) 327 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-15T22:29:04.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:04 smithi162 conmon[39755]: audit 2024-01-15T22:29:04.292947+0000 mon.smithi099 (mon.0) 324 : audit [INF] 2024-01-15T22:29:04.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:04 smithi162 conmon[39755]: from='client.? 172.21.15.99:0/1447421086' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c768cbd1-4c1d-4ba9-8273-a4b144d392ae"}]: dispatch 2024-01-15T22:29:04.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:04 smithi162 conmon[39755]: audit 2024-01-15T22:29:04.296568+0000 mon.smithi099 (mon.0) 325 : audit [INF] from='client.? 172.21.15.99:0/1447421086' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c768cbd1-4c1d-4ba9-8273-a4b144d392ae"}]': finished 2024-01-15T22:29:04.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:04 smithi162 conmon[39755]: cluster 2024-01-15T22:29:04.296611+0000 mon.smithi099 (mon.0) 326 : cluster [DBG] osdmap e16: 1 total, 0 up, 1 in 2024-01-15T22:29:04.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:04 smithi162 conmon[39755]: audit 2024-01-15T22:29:04.296725+0000 mon.smithi099 (mon.0) 327 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-15T22:29:05.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: cephadm 2024-01-15T22:29:04.206220+0000 mgr.smithi099.cmjeyn (mgr.14184) 77 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi162', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-15T22:29:05.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:29:05.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:29:05.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: return f(*arg) 2024-01-15T22:29:05.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:29:05.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:29:05.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:29:05.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: code, '\n'.join(err))) 2024-01-15T22:29:05.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:29:05.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpco61kzan:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp89e4iwv_:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:29:05.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:29:05.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:29:05.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:29:05.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 339f1c8c-0c2c-46f4-bd37-a13d21334cb9 2024-01-15T22:29:05.906 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:29:05.906 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:29:05.906 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:29:05.906 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:29:05.906 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:29:05.906 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr self.prepare() 2024-01-15T22:29:05.906 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:29:05.906 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:29:05.906 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:29:05.906 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:29:05.906 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:29:05.906 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:29:05.906 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:29:05.906 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:29:05.906 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:29:05.906 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:29:05.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:29:05.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:29:05.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:29:05.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:29:05.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:29:05.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:29:05.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:29:05.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:29:05.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:29:05.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:29:05.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:29:05.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:29:05.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:29:05.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:29:05.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:29:05.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:29:05.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:29:05.908 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:29:05.908 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:29:05.908 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:29:05.908 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:29:05.908 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:29:05.908 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:29:05.908 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:29:05.908 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:29:05.908 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:29:05.908 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:29:05.908 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:29:05.908 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:29:05.908 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:29:05.908 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:29:05.909 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:29:05.909 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:29:05.909 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:29:05.909 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:29:05.909 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:29:05.909 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:29:05.909 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:29:05.909 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:29:05.909 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:29:05.909 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:29:05.909 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:29:05.909 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: main() 2024-01-15T22:29:05.909 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:29:05.909 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: r = ctx.func(ctx) 2024-01-15T22:29:05.909 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:29:05.909 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:29:05.910 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:29:05.910 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:29:05.910 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:29:05.910 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:29:05.910 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774 2024-01-15T22:29:05.910 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:29:05.910 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:29:05.910 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:29:05.910 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:29:05.910 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:29:05.910 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:29:05.910 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpco61kzan:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp89e4iwv_:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:29:05.910 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: cluster 2024-01-15T22:29:04.492714+0000 mgr.smithi099.cmjeyn (mgr.14184) 78 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:05.910 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: audit 2024-01-15T22:29:04.610771+0000 mon.smithi099 (mon.0) 328 : audit [INF] from='client.? 172.21.15.99:0/2367121111' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:29:05.910 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: audit 2024-01-15T22:29:04.614512+0000 mon.smithi099 (mon.0) 329 : audit [INF] from='client.? 172.21.15.99:0/2367121111' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-15T22:29:05.911 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:05 smithi099 conmon[35292]: cluster 2024-01-15T22:29:04.614553+0000 mon.smithi099 (mon.0) 330 : cluster [DBG] osdmap e17: 0 total, 0 up, 0 in 2024-01-15T22:29:05.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: cephadm 2024-01-15T22:29:04.206220+0000 mgr.smithi099.cmjeyn (mgr.14184) 77 : cephadm [ERR] 2024-01-15T22:29:05.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi162', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-15T22:29:05.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:29:05.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:29:05.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: return f(*arg) 2024-01-15T22:29:05.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:29:05.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:29:05.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:29:05.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: code, '\n'.join(err))) 2024-01-15T22:29:05.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:29:05.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpco61kzan:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp89e4iwv_:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:29:05.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:29:05.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:29:05.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:29:05.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 339f1c8c-0c2c-46f4-bd37-a13d21334cb9 2024-01-15T22:29:05.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:29:05.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:29:05.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:29:05.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:29:05.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:29:05.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr self.prepare() 2024-01-15T22:29:05.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:29:05.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:29:05.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:29:05.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:29:05.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:29:05.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:29:05.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:29:05.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:29:05.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:29:05.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:29:05.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:29:05.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:29:05.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:29:05.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:29:05.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:29:05.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:29:05.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:29:05.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:29:05.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:29:05.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:29:05.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:29:05.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:29:05.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:29:05.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:29:05.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:29:05.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:29:05.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:29:05.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:29:05.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:29:05.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:29:05.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:29:05.951 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:29:05.951 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:29:05.951 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:29:05.951 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:29:05.951 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:29:05.951 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:29:05.951 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:29:05.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:29:05.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:29:05.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:29:05.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:29:05.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:29:05.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:29:05.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:29:05.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:29:05.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:29:05.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:29:05.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:29:05.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:29:05.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:29:05.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:29:05.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: main() 2024-01-15T22:29:05.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:29:05.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: r = ctx.func(ctx) 2024-01-15T22:29:05.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:29:05.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:29:05.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:29:05.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:29:05.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:29:05.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:29:05.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927 2024-01-15T22:29:05.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: , in _validate_fsid 2024-01-15T22:29:05.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:29:05.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:29:05.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:29:05.954 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:29:05.954 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:29:05.954 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpco61kzan:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp89e4iwv_:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:29:05.954 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: cluster 2024-01-15T22:29:04.492714+0000 mgr.smithi099.cmjeyn (mgr.14184) 78 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:05.954 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: audit 2024-01-15T22:29:04.610771+0000 mon.smithi099 (mon.0) 328 : audit [INF] from='client.? 172.21.15.99:0/2367121111' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:29:05.954 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: audit 2024-01-15T22:29:04.614512+0000 mon.smithi099 (mon.0) 329 : audit [INF] from='client.? 172.21.15.99:0/2367121111' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-15T22:29:05.954 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:05 smithi162 conmon[39755]: cluster 2024-01-15T22:29:04.614553+0000 mon.smithi099 (mon.0) 330 : cluster [DBG] osdmap e17: 0 total, 0 up, 0 in 2024-01-15T22:29:06.476 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:29:06.833 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: cephadm 2024-01-15T22:29:05.738600+0000 mgr.smithi099.cmjeyn (mgr.14184) 79 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi162', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-15T22:29:06.833 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:29:06.833 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:29:06.834 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: return f(*arg) 2024-01-15T22:29:06.834 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:29:06.834 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:29:06.834 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:29:06.834 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: code, '\n'.join(err))) 2024-01-15T22:29:06.834 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi099/config 2024-01-15T22:29:06.834 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnwm4v6xm:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp3get8u04:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:29:06.834 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:29:06.834 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:29:06.834 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:29:06.834 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new c768cbd1-4c1d-4ba9-8273-a4b144d392ae 2024-01-15T22:29:06.834 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:29:06.834 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:29:06.834 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:29:06.835 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:29:06.835 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:29:06.835 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr self.prepare() 2024-01-15T22:29:06.835 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:29:06.835 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:29:06.835 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:29:06.835 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:29:06.835 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:29:06.835 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:29:06.835 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:29:06.835 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:29:06.835 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:29:06.835 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:29:06.835 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:29:06.835 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:29:06.836 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:29:06.836 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:29:06.836 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:29:06.836 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:29:06.836 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:29:06.836 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:29:06.836 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:29:06.836 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:29:06.836 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:29:06.836 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:29:06.836 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:29:06.836 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:29:06.836 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:29:06.836 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:29:06.837 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:29:06.837 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:29:06.837 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:29:06.837 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:29:06.837 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:29:06.837 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:29:06.837 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:29:06.837 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:29:06.837 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:29:06.837 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:29:06.837 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:29:06.837 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:29:06.837 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:29:06.837 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:29:06.838 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:29:06.838 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:29:06.838 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:29:06.838 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:29:06.838 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:29:06.838 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:29:06.838 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:29:06.838 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:29:06.838 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:29:06.838 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:29:06.838 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:29:06.838 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:29:06.838 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: main() 2024-01-15T22:29:06.838 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:29:06.838 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: r = ctx.func(ctx) 2024-01-15T22:29:06.839 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:29:06.839 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:29:06.839 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:29:06.839 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:29:06.839 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:29:06.839 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:29:06.839 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774 2024-01-15T22:29:06.839 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:29:06.839 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:29:06.839 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:29:06.839 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:29:06.839 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:29:06.839 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:29:06.839 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnwm4v6xm:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp3get8u04:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:29:06.839 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: cephadm 2024-01-15T22:29:05.739761+0000 mgr.smithi099.cmjeyn (mgr.14184) 80 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-15T22:29:06.840 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: service_id: all-available-devices 2024-01-15T22:29:06.840 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: service_name: osd.all-available-devices 2024-01-15T22:29:06.840 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: placement: 2024-01-15T22:29:06.840 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: host_pattern: '*' 2024-01-15T22:29:06.840 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: spec: 2024-01-15T22:29:06.840 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: data_devices: 2024-01-15T22:29:06.840 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: all: true 2024-01-15T22:29:06.840 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: filter_logic: AND 2024-01-15T22:29:06.840 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: objectstore: bluestore 2024-01-15T22:29:06.840 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:29:06.840 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpco61kzan:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp89e4iwv_:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:29:06.840 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:29:06.840 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:29:06.840 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:29:06.841 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 339f1c8c-0c2c-46f4-bd37-a13d21334cb9 2024-01-15T22:29:06.841 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:29:06.841 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:29:06.841 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:29:06.841 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:29:06.841 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:29:06.841 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr self.prepare() 2024-01-15T22:29:06.841 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:29:06.841 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:29:06.841 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:29:06.841 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:29:06.841 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:29:06.841 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:29:06.841 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:29:06.842 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:29:06.842 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:29:06.842 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:29:06.842 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:29:06.842 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:29:06.842 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:29:06.842 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:29:06.842 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:29:06.842 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:29:06.842 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:29:06.842 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:29:06.842 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:29:06.842 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:29:06.842 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:29:06.842 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:29:06.842 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:29:06.843 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:29:06.843 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:29:06.843 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:29:06.843 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:29:06.843 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:29:06.843 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:29:06.843 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:29:06.843 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:29:06.843 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:29:06.843 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:29:06.843 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:29:06.843 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:29:06.843 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:29:06.843 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:29:06.843 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:29:06.844 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:29:06.844 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:29:06.844 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:29:06.844 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:29:06.844 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:29:06.844 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:29:06.844 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:29:06.844 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lv 2024-01-15T22:29:06.844 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: m/zap.py", line 302, in zap_osd 2024-01-15T22:29:06.844 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:29:06.844 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:29:06.844 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:29:06.844 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:29:06.844 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:29:06.844 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:29:06.845 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: main() 2024-01-15T22:29:06.845 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:29:06.845 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: r = ctx.func(ctx) 2024-01-15T22:29:06.845 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:29:06.845 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:29:06.845 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:29:06.845 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:29:06.846 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:29:06.846 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:29:06.846 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:29:06.846 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:29:06.846 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:29:06.846 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:29:06.846 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:29:06.846 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:29:06.846 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpco61kzan:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp89e4iwv_:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:29:06.846 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:29:06.846 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-15T22:29:06.846 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: if self._apply_service(spec): 2024-01-15T22:29:06.847 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-15T22:29:06.847 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-15T22:29:06.847 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-15T22:29:06.847 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-15T22:29:06.847 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-15T22:29:06.847 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-15T22:29:06.847 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-15T22:29:06.847 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-15T22:29:06.847 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-15T22:29:06.847 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: raise self._value 2024-01-15T22:29:06.847 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-15T22:29:06.847 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: result = (True, func(*args, **kwds)) 2024-01-15T22:29:06.847 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-15T22:29:06.847 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: return list(map(*args)) 2024-01-15T22:29:06.847 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:29:06.848 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: return f(*arg) 2024-01-15T22:29:06.848 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:29:06.848 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:29:06.848 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:29:06.848 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: code, '\n'.join(err))) 2024-01-15T22:29:06.848 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:29:06.848 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpco61kzan:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp89e4iwv_:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:29:06.848 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:29:06.848 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:29:06.848 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:29:06.848 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 339f1c8c-0c2c-46f4-bd37-a13d21334cb9 2024-01-15T22:29:06.848 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:29:06.848 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:29:06.848 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:29:06.848 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:29:06.848 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:29:06.849 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr self.prepare() 2024-01-15T22:29:06.849 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:29:06.849 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:29:06.849 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:29:06.849 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:29:06.849 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:29:06.849 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:29:06.849 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:29:06.849 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:29:06.849 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:29:06.849 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:29:06.849 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:29:06.849 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:29:06.849 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:29:06.850 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:29:06.850 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:29:06.850 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:29:06.850 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:29:06.850 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr F 2024-01-15T22:29:06.850 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: ile "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:29:06.850 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:29:06.850 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:29:06.850 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:29:06.850 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:29:06.850 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:29:06.850 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:29:06.850 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:29:06.850 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:29:06.850 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:29:06.850 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:29:06.851 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:29:06.851 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:29:06.851 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:29:06.851 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:29:06.851 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:29:06.851 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:29:06.851 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:29:06.851 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:29:06.851 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:29:06.851 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:29:06.851 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:29:06.851 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:29:06.851 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:29:06.851 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:29:06.851 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:29:06.852 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:29:06.852 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:29:06.852 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:29:06.852 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:29:06.852 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:29:06.852 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:29:06.852 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:29:06.852 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:29:06.852 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:29:06.852 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: main() 2024-01-15T22:29:06.852 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:29:06.852 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: r = ctx.func(ctx) 2024-01-15T22:29:06.852 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:29:06.852 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:29:06.853 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:29:06.853 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:29:06.853 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:29:06.853 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:29:06.853 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:29:06.853 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:29:06.853 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:29:06.853 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:29:06.853 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:29:06.853 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:29:06.853 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpco61kzan:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp89e4iwv_:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:29:06.853 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: cluster 2024-01-15T22:29:05.740228+0000 mgr.smithi099.cmjeyn (mgr.14184) 81 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:06.853 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:06 smithi099 conmon[35292]: audit 2024-01-15T22:29:06.477013+0000 mon.smithi099 (mon.0) 331 : audit [DBG] from='client.? 172.21.15.99:0/1938544841' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:29:06.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: cephadm 2024-01-15T22:29:05.738600+0000 mgr.smithi099.cmjeyn (mgr.14184) 79 : 2024-01-15T22:29:06.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi162', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-15T22:29:06.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:29:06.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:29:06.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: return f(*arg) 2024-01-15T22:29:06.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:29:06.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:29:06.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:29:06.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: code, '\n'.join(err))) 2024-01-15T22:29:06.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi099/config 2024-01-15T22:29:06.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnwm4v6xm:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp3get8u04:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:29:06.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:29:06.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:29:06.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:29:06.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new c768cbd1-4c1d-4ba9-8273-a4b144d392ae 2024-01-15T22:29:06.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:29:06.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:29:06.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:29:06.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:29:06.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:29:06.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr self.prepare() 2024-01-15T22:29:06.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:29:06.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:29:06.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:29:06.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:29:06.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:29:06.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:29:06.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:29:06.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:29:06.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:29:06.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:29:06.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:29:06.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:29:06.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:29:06.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:29:06.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:29:06.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:29:06.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:29:06.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:29:06.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:29:06.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:29:06.951 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:29:06.951 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:29:06.951 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:29:06.951 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:29:06.951 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:29:06.951 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:29:06.951 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:29:06.951 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:29:06.951 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:29:06.951 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:29:06.951 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:29:06.951 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:29:06.951 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:29:06.951 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:29:06.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:29:06.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:29:06.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:29:06.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:29:06.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:29:06.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:29:06.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:29:06.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:29:06.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:29:06.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:29:06.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:29:06.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:29:06.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:29:06.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:29:06.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:29:06.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:29:06.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:29:06.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:29:06.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: main() 2024-01-15T22:29:06.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:29:06.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: r = ctx.func(ctx) 2024-01-15T22:29:06.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:29:06.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:29:06.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:29:06.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:29:06.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:29:06.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:29:06.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90 2024-01-15T22:29:06.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: d2", line 1927, in _validate_fsid 2024-01-15T22:29:06.954 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:29:06.954 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:29:06.954 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:29:06.954 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:29:06.954 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:29:06.954 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnwm4v6xm:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp3get8u04:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:29:06.954 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: cephadm 2024-01-15T22:29:05.739761+0000 mgr.smithi099.cmjeyn (mgr.14184) 80 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-15T22:29:06.954 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: service_id: all-available-devices 2024-01-15T22:29:06.954 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: service_name: osd.all-available-devices 2024-01-15T22:29:06.954 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: placement: 2024-01-15T22:29:06.954 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: host_pattern: '*' 2024-01-15T22:29:06.954 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: spec: 2024-01-15T22:29:06.954 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: data_devices: 2024-01-15T22:29:06.954 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: all: true 2024-01-15T22:29:06.954 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: filter_logic: AND 2024-01-15T22:29:06.955 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: objectstore: bluestore 2024-01-15T22:29:06.955 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:29:06.955 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpco61kzan:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp89e4iwv_:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:29:06.955 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:29:06.955 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:29:06.955 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:29:06.955 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 339f1c8c-0c2c-46f4-bd37-a13d21334cb9 2024-01-15T22:29:06.955 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:29:06.955 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:29:06.955 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:29:06.955 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:29:06.955 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:29:06.955 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr self.prepare() 2024-01-15T22:29:06.955 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:29:06.956 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:29:06.956 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:29:06.956 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:29:06.956 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:29:06.956 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:29:06.956 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:29:06.956 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:29:06.956 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:29:06.956 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:29:06.956 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:29:06.956 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:29:06.956 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:29:06.956 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:29:06.956 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:29:06.956 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:29:06.957 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:29:06.957 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:29:06.957 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:29:06.957 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:29:06.957 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:29:06.957 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:29:06.957 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:29:06.957 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:29:06.957 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:29:06.957 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:29:06.958 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:29:06.958 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:29:06.958 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:29:06.958 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:29:06.958 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:29:06.958 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:29:06.958 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:29:06.959 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:29:06.959 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:29:06.959 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:29:06.959 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:29:06.959 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:29:06.959 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:29:06.959 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:29:06.959 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:29:06.959 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:29:06.959 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:29:06.959 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:29:06.959 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:29:06.959 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:29:06.959 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr devices = find_associa 2024-01-15T22:29:06.960 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: ted_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:29:06.960 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:29:06.960 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:29:06.960 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:29:06.960 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:29:06.960 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:29:06.960 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: main() 2024-01-15T22:29:06.960 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:29:06.960 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: r = ctx.func(ctx) 2024-01-15T22:29:06.960 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:29:06.960 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:29:06.960 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:29:06.960 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:29:06.960 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:29:06.961 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:29:06.961 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:29:06.961 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:29:06.961 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:29:06.961 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:29:06.961 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:29:06.961 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:29:06.961 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpco61kzan:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp89e4iwv_:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:29:06.961 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:29:06.961 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-15T22:29:06.961 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: if self._apply_service(spec): 2024-01-15T22:29:06.961 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-15T22:29:06.961 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-15T22:29:06.961 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-15T22:29:06.962 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-15T22:29:06.962 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-15T22:29:06.962 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-15T22:29:06.962 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-15T22:29:06.962 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-15T22:29:06.962 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-15T22:29:06.962 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: raise self._value 2024-01-15T22:29:06.962 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-15T22:29:06.962 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: result = (True, func(*args, **kwds)) 2024-01-15T22:29:06.962 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-15T22:29:06.962 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: return list(map(*args)) 2024-01-15T22:29:06.962 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:29:06.962 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: return f(*arg) 2024-01-15T22:29:06.962 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:29:06.963 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:29:06.963 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:29:06.963 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: code, '\n'.join(err))) 2024-01-15T22:29:06.963 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:29:06.963 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpco61kzan:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp89e4iwv_:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:29:06.963 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:29:06.963 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:29:06.963 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:29:06.963 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 339f1c8c-0c2c-46f4-bd37-a13d21334cb9 2024-01-15T22:29:06.963 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:29:06.963 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:29:06.963 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:29:06.963 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:29:06.963 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:29:06.963 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr self.prepare() 2024-01-15T22:29:06.964 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:29:06.964 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:29:06.964 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:29:06.964 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:29:06.964 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:29:06.964 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:29:06.964 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:29:06.964 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:29:06.964 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:29:06.964 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:29:06.964 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:29:06.964 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:29:06.964 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:29:06.964 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:29:06.965 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:29:06.965 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:29:06.965 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:29:06.965 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in 2024-01-15T22:29:06.965 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: newfunc 2024-01-15T22:29:06.965 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:29:06.965 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:29:06.965 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:29:06.965 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:29:06.965 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:29:06.965 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:29:06.965 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:29:06.965 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:29:06.965 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:29:06.965 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:29:06.965 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:29:06.965 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:29:06.966 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:29:06.966 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:29:06.966 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:29:06.966 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:29:06.966 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:29:06.966 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:29:06.966 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:29:06.966 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:29:06.966 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:29:06.966 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:29:06.966 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:29:06.966 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:29:06.966 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:29:06.966 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:29:06.966 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:29:06.967 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:29:06.967 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:29:06.967 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:29:06.967 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:29:06.967 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:29:06.967 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:29:06.967 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:29:06.967 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: main() 2024-01-15T22:29:06.967 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:29:06.967 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: r = ctx.func(ctx) 2024-01-15T22:29:06.967 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:29:06.967 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:29:06.967 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:29:06.967 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:29:06.967 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:29:06.967 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:29:06.968 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:29:06.968 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:29:06.968 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:29:06.968 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:29:06.968 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:29:06.968 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:29:06.968 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpco61kzan:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp89e4iwv_:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:29:06.968 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: cluster 2024-01-15T22:29:05.740228+0000 mgr.smithi099.cmjeyn (mgr.14184) 81 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:06.970 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:06 smithi162 conmon[39755]: audit 2024-01-15T22:29:06.477013+0000 mon.smithi099 (mon.0) 331 : audit [DBG] from='client.? 172.21.15.99:0/1938544841' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:29:07.305 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:29:07.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:07 smithi099 conmon[35292]: cluster 2024-01-15T22:29:06.501701+0000 mon.smithi099 (mon.0) 332 : cluster [WRN] 2024-01-15T22:29:07.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:07 smithi099 conmon[35292]: Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-15T22:29:07.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:07 smithi162 conmon[39755]: cluster 2024-01-15T22:29:06.501701+0000 mon.smithi099 (mon.0) 332 : cluster [WRN] 2024-01-15T22:29:07.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:07 smithi162 conmon[39755]: Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-15T22:29:08.306 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:29:08.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:08 smithi099 conmon[35292]: cluster 2024-01-15T22:29:07.740868+0000 mgr.smithi099.cmjeyn (mgr.14184) 82 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:08.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:08 smithi162 conmon[39755]: cluster 2024-01-15T22:29:07.740868+0000 mgr.smithi099.cmjeyn (mgr.14184) 82 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:10.020 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:29:10.597 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:29:10.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:10 smithi099 conmon[35292]: cluster 2024-01-15T22:29:09.741219+0000 mgr.smithi099.cmjeyn (mgr.14184) 83 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:10.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:10 smithi099 conmon[35292]: 2024-01-15T22:29:10.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:10 smithi099 conmon[35292]: audit 2024-01-15T22:29:10.021324+0000 mon.smithi099 (mon.0) 333 : audit [DBG] from='client.? 172.21.15.99:0/414538097' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:29:11.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:10 smithi162 conmon[39755]: cluster 2024-01-15T22:29:09.741219+0000 mgr.smithi099.cmjeyn (mgr.14184) 83 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:11.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:10 smithi162 conmon[39755]: audit 2024-01-15T22:29:10.021324+0000 mon.smithi099 (mon.0) 333 : audit [DBG] from='client.? 172.21.15.99:0/414538097' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:29:11.598 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:29:11.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:11 smithi099 conmon[35292]: cluster 2024-01-15T22:29:11.741577+0000 mgr.smithi099.cmjeyn (mgr.14184) 84 : 2024-01-15T22:29:11.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:11 smithi099 conmon[35292]: cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:12.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:11 smithi162 conmon[39755]: cluster 2024-01-15T22:29:11.741577+0000 mgr.smithi099.cmjeyn (mgr.14184) 84 : cluster [DBG] 2024-01-15T22:29:12.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:11 smithi162 conmon[39755]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:13.322 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:29:13.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:13 smithi099 conmon[35292]: audit 2024-01-15T22:29:13.323048+0000 mon.smithi099 ( 2024-01-15T22:29:13.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:13 smithi099 conmon[35292]: mon.0) 334 : audit [DBG] from='client.? 172.21.15.99:0/722282928' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:29:13.696 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:13 smithi162 conmon[39755]: audit 2024-01-15T22:29:13.323048+0000 mon.smithi099 (mon.0) 334 : audit 2024-01-15T22:29:13.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:13 smithi162 conmon[39755]: [DBG] from='client.? 172.21.15.99:0/722282928' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:29:13.923 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:29:14.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:14 smithi099 conmon[35292]: cluster 2024-01-15T22:29:13.742218+0000 mgr.smithi099.cmjeyn (mgr.14184) 85 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:14.696 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:14 smithi162 conmon[39755]: cluster 2024-01-15T22:29:13.742218+0000 mgr.smithi099.cmjeyn (mgr.14184) 85 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:14.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:14 smithi162 conmon[39755]: 2024-01-15T22:29:14.925 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:29:16.095 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:15 smithi099 conmon[35292]: cluster 2024-01-15T22:29:15.742630+0000 mgr.smithi099.cmjeyn (mgr.14184) 86 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:16.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:15 smithi162 conmon[39755]: cluster 2024-01-15T22:29:15.742630+0000 mgr.smithi099.cmjeyn (mgr.14184) 86 : cluster [DBG] 2024-01-15T22:29:16.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:15 smithi162 conmon[39755]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:16.690 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:29:17.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:16 smithi099 conmon[35292]: audit 2024-01-15T22:29:16.690702+0000 mon.smithi099 (mon.0) 2024-01-15T22:29:17.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:16 smithi099 conmon[35292]: 335 : audit [DBG] from='client.? 172.21.15.99:0/936155416' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:29:17.196 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:16 smithi162 conmon[39755]: audit 2024-01-15T22:29:16.690702+0000 mon.smithi099 (mon.0) 2024-01-15T22:29:17.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:16 smithi162 conmon[39755]: 335 : audit [DBG] from='client.? 172.21.15.99:0/936155416' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:29:17.218 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:29:18.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:17 smithi162 conmon[39755]: cluster 2024-01-15T22:29:17.743275+0000 mgr.smithi099.cmjeyn (mgr.14184) 87 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:18.219 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:29:18.251 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:17 smithi099 conmon[35292]: cluster 2024-01-15T22:29:17.743275+0000 mgr.smithi099.cmjeyn (mgr.14184) 87 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:19.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:19 smithi162 conmon[39755]: cluster 2024-01-15T22:29:19.743686+0000 mgr.smithi099.cmjeyn (mgr.14184) 88 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:19.980 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:29:20.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:19 smithi099 conmon[35292]: cluster 2024-01-15T22:29:19.743686+0000 mgr.smithi099.cmjeyn (mgr.14184) 88 2024-01-15T22:29:20.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:19 smithi099 conmon[35292]: : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:20.585 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:29:21.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:20 smithi162 conmon[39755]: audit 2024-01-15T22:29:19.980874+0000 mon.smithi099 (mon.0) 336 : audit [DBG] from='client.? 172.21.15.99:0/805472389' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:29:21.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:20 smithi099 conmon[35292]: audit 2024-01-15T22:29:19.980874+0000 mon.smithi099 (mon.0) 336 : audit 2024-01-15T22:29:21.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:20 smithi099 conmon[35292]: [DBG] from='client.? 172.21.15.99:0/805472389' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:29:21.586 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:29:22.176 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:21 smithi099 conmon[35292]: cluster 2024-01-15T22:29:21.744085+0000 mgr.smithi099.cmjeyn (mgr.14184) 89 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:22.196 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:21 smithi162 conmon[39755]: cluster 2024-01-15T22:29:21.744085+0000 mgr.smithi099.cmjeyn (mgr.14184) 89 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:23.313 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:29:23.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:23 smithi099 conmon[35292]: audit 2024-01-15T22:29:23.313936+0000 mon.smithi099 (mon.0) 337 2024-01-15T22:29:23.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:23 smithi099 conmon[35292]: : audit [DBG] from='client.? 172.21.15.99:0/1987432955' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:29:23.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:23 smithi162 conmon[39755]: audit 2024-01-15T22:29:23.313936+0000 mon.smithi099 (mon.0) 337 : audit [DBG] from='client.? 172.21.15.99:0/1987432955' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:29:23.901 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:29:24.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:24 smithi099 conmon[35292]: cluster 2024-01-15T22:29:23.744461+0000 mgr.smithi099.cmjeyn (mgr.14184 2024-01-15T22:29:24.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:24 smithi099 conmon[35292]: ) 90 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:24.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:24 smithi162 conmon[39755]: cluster 2024-01-15T22:29:23.744461+0000 mgr.smithi099.cmjeyn (mgr.14184) 90 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:24.902 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:29:26.126 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:25 smithi099 conmon[35292]: cluster 2024-01-15T22:29:25.745063+0000 mgr.smithi099.cmjeyn (mgr.14184) 91 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:26.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:25 smithi162 conmon[39755]: cluster 2024-01-15T22:29:25.745063+0000 mgr.smithi099.cmjeyn (mgr.14184) 91 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:26.707 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:29:27.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:26 smithi099 conmon[35292]: audit 2024-01-15T22:29:26.707730+0000 mon.smithi099 (mon.0) 338 : audit [DBG] 2024-01-15T22:29:27.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:26 smithi099 conmon[35292]: from='client.? 172.21.15.99:0/1644370258' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:29:27.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:26 smithi162 conmon[39755]: audit 2024-01-15T22:29:26.707730+0000 mon.smithi099 (mon.0) 338 : audit [DBG] from='client.? 172.21.15.99:0/1644370258' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:29:27.267 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:29:28.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:27 smithi162 conmon[39755]: cluster 2024-01-15T22:29:27.745616+0000 mgr.smithi099.cmjeyn (mgr.14184) 92 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:28.268 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:29:28.302 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:27 smithi099 conmon[35292]: cluster 2024-01-15T22:29:27.745616+0000 mgr.smithi099.cmjeyn (mgr.14184) 92 : cluster 2024-01-15T22:29:28.303 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:27 smithi099 conmon[35292]: [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:29.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:29 smithi162 conmon[39755]: cluster 2024-01-15T22:29:29.746008+0000 mgr.smithi099.cmjeyn (mgr.14184) 93 : cluster [DBG] 2024-01-15T22:29:29.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:29 smithi162 conmon[39755]: pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:30.024 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:29:30.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:29 smithi099 conmon[35292]: cluster 2024-01-15T22:29:29.746008+0000 mgr.smithi099.cmjeyn (mgr.14184) 93 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:30.566 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:29:31.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:30 smithi162 conmon[39755]: audit 2024-01-15T22:29:30.024485+0000 mon.smithi099 (mon.0) 339 : audit [DBG] from='client.? 172.21.15.99:0/4138945048' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:29:31.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:30 smithi099 conmon[35292]: audit 2024-01-15T22:29:30.024485+0000 mon.smithi099 (mon.0) 339 : audit [DBG] 2024-01-15T22:29:31.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:30 smithi099 conmon[35292]: from='client.? 172.21.15.99:0/4138945048' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:29:31.567 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:29:32.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:31 smithi162 conmon[39755]: cluster 2024-01-15T22:29:31.746374+0000 mgr.smithi099.cmjeyn (mgr.14184) 94 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:32.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:31 smithi162 conmon[39755]: 2024-01-15T22:29:32.335 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:31 smithi099 conmon[35292]: cluster 2024-01-15T22:29:31.746374+0000 mgr.smithi099.cmjeyn (mgr.14184) 94 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:33.335 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:29:33.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:33 smithi099 conmon[35292]: audit 2024-01-15T22:29:33.335260+0000 mon.smithi099 (mon. 2024-01-15T22:29:33.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:33 smithi099 conmon[35292]: 0) 340 : audit [DBG] from='client.? 172.21.15.99:0/965993524' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:29:33.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:33 smithi162 conmon[39755]: audit 2024-01-15T22:29:33.335260+0000 mon.smithi099 (mon.0) 340 : audit [DBG] 2024-01-15T22:29:33.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:33 smithi162 conmon[39755]: from='client.? 172.21.15.99:0/965993524' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:29:33.900 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:29:34.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:34 smithi099 conmon[35292]: cluster 2024-01-15T22:29:33.746714+0000 mgr.smithi099.cmjeyn (mgr.14184) 95 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:34.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:34 smithi099 conmon[35292]: 2024-01-15T22:29:34.696 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:34 smithi162 conmon[39755]: cluster 2024-01-15T22:29:33.746714+0000 mgr.smithi099.cmjeyn (mgr.14184) 95 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:34.902 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:29:36.092 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:35 smithi099 conmon[35292]: cluster 2024-01-15T22:29:35.747073+0000 mgr.smithi099.cmjeyn (mgr.14184) 96 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:36.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:35 smithi162 conmon[39755]: cluster 2024-01-15T22:29:35.747073+0000 mgr.smithi099.cmjeyn (mgr.14184) 96 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:36.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:35 smithi162 conmon[39755]: 2024-01-15T22:29:36.624 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:29:37.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:36 smithi162 conmon[39755]: audit 2024-01-15T22:29:36.625093+0000 mon.smithi099 (mon.0) 341 : audit [DBG] 2024-01-15T22:29:37.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:36 smithi162 conmon[39755]: from='client.? 172.21.15.99:0/322756905' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:29:37.234 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:36 smithi099 conmon[35292]: audit 2024-01-15T22:29:36.625093+0000 mon.smithi099 (mon.0) 341 : 2024-01-15T22:29:37.234 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:36 smithi099 conmon[35292]: audit [DBG] from='client.? 172.21.15.99:0/322756905' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:29:37.248 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:29:38.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:37 smithi162 conmon[39755]: cluster 2024-01-15T22:29:37.747706+0000 mgr.smithi099.cmjeyn (mgr.14184) 97 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:38.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:37 smithi162 conmon[39755]: 2024-01-15T22:29:38.249 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:29:38.282 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:37 smithi099 conmon[35292]: cluster 2024-01-15T22:29:37.747706+0000 mgr.smithi099.cmjeyn (mgr.14184) 97 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:39.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:39 smithi162 conmon[39755]: cluster 2024-01-15T22:29:39.748069+0000 mgr.smithi099.cmjeyn (mgr.14184) 98 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:39.965 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:29:40.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:39 smithi099 conmon[35292]: cluster 2024-01-15T22:29:39.748069+0000 mgr.smithi099.cmjeyn (mgr.14184) 98 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:40.541 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:29:41.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:40 smithi162 conmon[39755]: audit 2024-01-15T22:29:39.964820+0000 mon.smithi099 (mon.0) 342 : audit [DBG] from='client.? 172.21.15.99:0/1568997593' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:29:41.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:40 smithi099 conmon[35292]: audit 2024-01-15T22:29:39.964820+0000 mon.smithi099 (mon.0) 342 2024-01-15T22:29:41.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:40 smithi099 conmon[35292]: : audit [DBG] from='client.? 172.21.15.99:0/1568997593' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:29:41.542 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:29:42.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:41 smithi162 conmon[39755]: cluster 2024-01-15T22:29:41.748441+0000 mgr.smithi099.cmjeyn (mgr.14184) 99 : cluster [DBG] 2024-01-15T22:29:42.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:41 smithi162 conmon[39755]: pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:42.339 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:41 smithi099 conmon[35292]: cluster 2024-01-15T22:29:41.748441+0000 mgr.smithi099.cmjeyn (mgr.14184) 99 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:43.319 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:29:43.598 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:43 smithi099 conmon[35292]: audit 2024-01-15T22:29:43.319671+0000 mon.smithi099 (mon.0) 343 : audit [DBG] from='client.? 172.21.15.99:0/333579115' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:29:43.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:43 smithi162 conmon[39755]: audit 2024-01-15T22:29:43.319671+0000 mon.smithi099 (mon.0) 343 : audit [DBG] 2024-01-15T22:29:43.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:43 smithi162 conmon[39755]: from='client.? 172.21.15.99:0/333579115' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:29:43.965 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:29:44.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:44 smithi099 conmon[35292]: cluster 2024-01-15T22:29:43.748773+0000 mgr.smithi099.cmjeyn (mgr.14184) 100 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:44.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:44 smithi162 conmon[39755]: cluster 2024-01-15T22:29:43.748773+0000 mgr.smithi099.cmjeyn (mgr.14184) 100 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:44.966 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:29:46.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:45 smithi099 conmon[35292]: cluster 2024-01-15T22:29:45.749154+0000 mgr.smithi099.cmjeyn (mgr.14184) 101 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:46.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:45 smithi162 conmon[39755]: cluster 2024-01-15T22:29:45.749154+0000 mgr.smithi099.cmjeyn (mgr.14184) 101 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:46.730 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:29:47.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:46 smithi099 conmon[35292]: audit 2024-01-15T22:29:46.731292+0000 mon.smithi099 (mon.0) 344 2024-01-15T22:29:47.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:46 smithi099 conmon[35292]: : audit [DBG] from='client.? 172.21.15.99:0/1095835201' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:29:47.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:46 smithi162 conmon[39755]: audit 2024-01-15T22:29:46.731292+0000 mon.smithi099 (mon.0) 344 : audit [DBG] from='client.? 172.21.15.99:0/1095835201' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:29:47.313 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:29:48.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:47 smithi162 conmon[39755]: cluster 2024-01-15T22:29:47.749539+0000 mgr.smithi099.cmjeyn (mgr.14184) 102 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:48.314 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:29:48.346 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:47 smithi099 conmon[35292]: cluster 2024-01-15T22:29:47.749539+0000 mgr.smithi099.cmjeyn (mgr.14184) 102 : 2024-01-15T22:29:48.346 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:47 smithi099 conmon[35292]: cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:49.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:49 smithi162 conmon[39755]: cluster 2024-01-15T22:29:49.750060+0000 mgr.smithi099.cmjeyn (mgr.14184) 103 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:50.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:49 smithi099 conmon[35292]: cluster 2024-01-15T22:29:49.750060+0000 mgr.smithi099.cmjeyn (mgr.14184) 103 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:50.194 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:29:50.764 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:29:51.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:50 smithi099 conmon[35292]: audit 2024-01-15T22:29:50.195053+0000 mon.smithi099 (mon.0) 345 : audit [DBG] from='client.? 172.21.15.99:0/2216920056' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:29:51.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:50 smithi162 conmon[39755]: audit 2024-01-15T22:29:50.195053+0000 mon.smithi099 (mon.0) 345 : audit [DBG] from='client.? 172.21.15.99:0/2216920056' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:29:51.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:50 smithi162 conmon[39755]: 2024-01-15T22:29:51.765 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:29:52.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:51 smithi099 conmon[35292]: cluster 2024-01-15T22:29:51.750539+0000 mgr.smithi099.cmjeyn (mgr.14184) 104 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:52.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:51 smithi162 conmon[39755]: cluster 2024-01-15T22:29:51.750539+0000 mgr.smithi099.cmjeyn (mgr.14184) 104 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:53.542 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:29:53.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:53 smithi099 conmon[35292]: audit 2024-01-15T22:29:53.542672+0000 mon.smithi099 (mon.0) 346 : audit [DBG] from='client.? 172.21.15.99:0/1796534979' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:29:53.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:53 smithi162 conmon[39755]: audit 2024-01-15T22:29:53.542672+0000 mon.smithi099 (mon.0) 346 : audit [DBG] from='client.? 172.21.15.99:0/1796534979' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:29:54.160 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:29:54.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:54 smithi099 conmon[35292]: cluster 2024-01-15T22:29:53.751006+0000 mgr.smithi099.cmjeyn (mgr.14184) 105 : cluster [DBG] 2024-01-15T22:29:54.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:54 smithi099 conmon[35292]: pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:54.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:54 smithi162 conmon[39755]: cluster 2024-01-15T22:29:53.751006+0000 mgr.smithi099.cmjeyn (mgr.14184) 105 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:55.162 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:29:56.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:55 smithi162 conmon[39755]: cluster 2024-01-15T22:29:55.751378+0000 mgr.smithi099.cmjeyn (mgr.14184) 106 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:56.345 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:55 smithi099 conmon[35292]: cluster 2024-01-15T22:29:55.751378+0000 mgr.smithi099.cmjeyn (mgr.14184) 106 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:56.883 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:29:57.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:56 smithi162 conmon[39755]: audit 2024-01-15T22:29:56.883643+0000 mon.smithi099 (mon.0) 347 : audit [DBG] from='client.? 172.21.15.99:0/3055335513' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:29:57.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:56 smithi099 conmon[35292]: audit 2024-01-15T22:29:56.883643+0000 mon.smithi099 (mon.0) 347 2024-01-15T22:29:57.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:56 smithi099 conmon[35292]: : audit [DBG] from='client.? 172.21.15.99:0/3055335513' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:29:57.511 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:29:58.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:57 smithi162 conmon[39755]: cluster 2024-01-15T22:29:57.751988+0000 mgr.smithi099.cmjeyn (mgr.14184) 107 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:58.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:57 smithi099 conmon[35292]: cluster 2024-01-15T22:29:57.751988+0000 mgr.smithi099.cmjeyn (mgr.14184) 107 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:58.512 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:29:59.922 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:59 smithi099 conmon[35292]: cluster 2024-01-15T22:29:59.752539+0000 mgr.smithi099.cmjeyn (mgr.14184) 2024-01-15T22:29:59.922 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:29:59 smithi099 conmon[35292]: 108 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:29:59.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:29:59 smithi162 conmon[39755]: cluster 2024-01-15T22:29:59.752539+0000 mgr.smithi099.cmjeyn (mgr.14184) 108 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:00.214 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:30:00.802 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:30:01.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:00.000149+0000 mon.smithi099 (mon.0) 348 : cluster [WRN] Health detail: HEALTH_WARN Failed to apply 1 service(s): osd.all-available-devices 2024-01-15T22:30:01.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:00.000175+0000 mon.smithi099 (mon.0) 349 : cluster [WRN] [WRN] CEPHADM_APPLY_SPEC_FAIL: Failed to apply 1 service(s): osd.all-available-devices 2024-01-15T22:30:01.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:00.000191+0000 mon.smithi099 (mon.0) 350 : cluster [WRN] osd.all-available-devices: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:30:01.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:00.000208+0000 mon.smithi099 (mon.0) 351 : cluster [WRN] Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpco61kzan:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp89e4iwv_:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:30:01.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:00.000220+0000 mon.smithi099 (mon.0) 352 : cluster [WRN] /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:30:01.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:00.000232+0000 mon.smithi099 (mon.0) 353 : cluster [WRN] /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:30:01.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:00.000245+0000 mon.smithi099 (mon.0) 354 : cluster [WRN] /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:30:01.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:00.000259+0000 2024-01-15T22:30:01.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: mon.smithi099 (mon.0) 355 : cluster [WRN] /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 339f1c8c-0c2c-46f4-bd37-a13d21334cb9 2024-01-15T22:30:01.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:00.000270+0000 mon.smithi099 (mon.0) 356 : cluster [WRN] /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:30:01.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:00.000283+0000 mon.smithi099 (mon.0) 357 : cluster [WRN] /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:30:01.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:00.000294+0000 mon.smithi099 (mon.0) 358 : cluster [WRN] /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:30:01.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:00.000305+0000 mon.smithi099 (mon.0) 359 : cluster [WRN] /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:30:01.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:00.000317+0000 mon.smithi099 (mon.0) 360 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:30:01.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:00.000337+0000 mon.smithi099 (mon.0) 361 : cluster [WRN] /bin/podman: stderr self.prepare() 2024-01-15T22:30:01.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:00.000349+0000 mon.smithi099 (mon.0) 362 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:01.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:00.000361+0000 mon.smithi099 (mon.0) 363 : cluster [WRN] /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:01.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:00.000373+0000 mon.smithi099 (mon.0) 364 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:30:01.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:00.000385+0000 mon.smithi099 (mon.0) 365 : cluster [WRN] /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:30:01.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:00.000397+0000 mon.smithi099 (mon.0) 366 : 2024-01-15T22:30:01.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:30:01.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:00.000409+0000 mon.smithi099 (mon.0) 367 : cluster [WRN] /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:30:01.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:00.000420+0000 mon.smithi099 ( 2024-01-15T22:30:01.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: mon.0) 368 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:30:01.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024- 2024-01-15T22:30:01.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 01-15T22:30:00.000431+0000 mon.smithi099 (mon.0) 369 2024-01-15T22:30:01.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: : cluster [WRN] /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:30:01.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:00 2024-01-15T22:30:01.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: .000442+0000 mon.smithi099 (mon.0) 370 : cluster 2024-01-15T22:30:01.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: [WRN] /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:30:01.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:01.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 2024-01-15T22:30:00.000453+0000 mon.smithi099 (mon. 2024-01-15T22:30:01.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 0) 371 : cluster [WRN] /bin/podman: stderr 2024-01-15T22:30:01.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22 2024-01-15T22:30:01.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: :30:00.000469+0000 mon.smithi099 (mon.0) 372 2024-01-15T22:30:01.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: : cluster [WRN] /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:30:01.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:00 2024-01-15T22:30:01.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: .000481+0000 mon.smithi099 (mon.0) 373 : cluster 2024-01-15T22:30:01.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: [WRN] /bin/podman: stderr 2024-01-15T22:30:01.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:01.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 2024-01-15T22:30:00.000493+0000 mon.smithi099 ( 2024-01-15T22:30:01.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: mon.0) 374 : cluster [WRN] /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:30:01.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15 2024-01-15T22:30:01.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: T22:30:00.000504+0000 mon.smithi099 (mon. 2024-01-15T22:30:01.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 0) 375 : cluster [WRN] /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:30:01.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22: 2024-01-15T22:30:01.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 30:00.000516+0000 mon.smithi099 (mon.0) 376 2024-01-15T22:30:01.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: : cluster [WRN] /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:30:01.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:00. 2024-01-15T22:30:01.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 000527+0000 mon.smithi099 (mon.0) 377 : cluster [WRN] 2024-01-15T22:30:01.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:30:01.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024 2024-01-15T22:30:01.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: -01-15T22:30:00.000539+0000 mon.smithi099 (mon.0 2024-01-15T22:30:01.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: ) 378 : cluster [WRN] /bin/podman: stderr self.main(self.argv) 2024-01-15T22:30:01.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30 2024-01-15T22:30:01.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: :00.000550+0000 mon.smithi099 (mon.0) 379 : cluster 2024-01-15T22:30:01.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:30:01.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:01.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 2024-01-15T22:30:00.000561+0000 mon.smithi099 ( 2024-01-15T22:30:01.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: mon.0) 380 : cluster [WRN] /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:30:01.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15 2024-01-15T22:30:01.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: T22:30:00.000573+0000 mon.smithi099 (mon.0 2024-01-15T22:30:01.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: ) 381 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:30:01.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30: 2024-01-15T22:30:01.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 00.000586+0000 mon.smithi099 (mon.0) 382 : cluster 2024-01-15T22:30:01.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: [WRN] /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:30:01.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:01.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 2024-01-15T22:30:00.000597+0000 mon.smithi099 ( 2024-01-15T22:30:01.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: mon.0) 383 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:30:01.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22 2024-01-15T22:30:01.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: :30:00.000609+0000 mon.smithi099 (mon.0 2024-01-15T22:30:01.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: ) 384 : cluster [WRN] /bin/podman: stderr instance.main() 2024-01-15T22:30:01.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30 2024-01-15T22:30:01.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: :00.000620+0000 mon.smithi099 (mon.0) 385 2024-01-15T22:30:01.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:30:01.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30: 2024-01-15T22:30:01.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 00.000632+0000 mon.smithi099 (mon.0) 386 : cluster 2024-01-15T22:30:01.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: [WRN] /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:30:01.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:00 2024-01-15T22:30:01.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: .000642+0000 mon.smithi099 (mon.0) 387 : cluster 2024-01-15T22:30:01.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:30:01.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:01.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 2024-01-15T22:30:00.000652+0000 mon.smithi099 (mon 2024-01-15T22:30:01.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: .0) 388 : cluster [WRN] /bin/podman: stderr instance.main() 2024-01-15T22:30:01.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T 2024-01-15T22:30:01.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 22:30:00.000664+0000 mon.smithi099 (mon.0 2024-01-15T22:30:01.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: ) 389 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:01.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30 2024-01-15T22:30:01.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: :00.000679+0000 mon.smithi099 (mon.0) 390 2024-01-15T22:30:01.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: : cluster [WRN] /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:01.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:00 2024-01-15T22:30:01.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: .000691+0000 mon.smithi099 (mon.0) 391 : cluster 2024-01-15T22:30:01.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:30:01.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:01.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 2024-01-15T22:30:00.000703+0000 mon.smithi099 ( 2024-01-15T22:30:01.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: mon.0) 392 : cluster [WRN] /bin/podman: stderr self._execute(plan) 2024-01-15T22:30:01.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15 2024-01-15T22:30:01.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: T22:30:00.000714+0000 mon.smithi099 (mon 2024-01-15T22:30:01.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: .0) 393 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:30:01.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22 2024-01-15T22:30:01.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: :30:00.000726+0000 mon.smithi099 (mon.0) 394 2024-01-15T22:30:01.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: : cluster [WRN] /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:30:01.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:00 2024-01-15T22:30:01.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: .000737+0000 mon.smithi099 (mon.0) 395 : cluster [WRN] 2024-01-15T22:30:01.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:01.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:01.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 2024-01-15T22:30:00.000748+0000 mon.smithi099 (mon 2024-01-15T22:30:01.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: .0) 396 : cluster [WRN] /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:01.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22 2024-01-15T22:30:01.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: :30:00.000760+0000 mon.smithi099 (mon.0 2024-01-15T22:30:01.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: ) 397 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:30:01.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22: 2024-01-15T22:30:01.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 30:00.000771+0000 mon.smithi099 (mon.0) 2024-01-15T22:30:01.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 398 : cluster [WRN] /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:30:01.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30 2024-01-15T22:30:01.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: :00.000792+0000 mon.smithi099 (mon.0) 399 : 2024-01-15T22:30:01.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:30:01.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:00 2024-01-15T22:30:01.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: .000803+0000 mon.smithi099 (mon.0) 400 : cluster 2024-01-15T22:30:01.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: [WRN] /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:30:01.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:01.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 2024-01-15T22:30:00.000815+0000 mon.smithi099 (mon 2024-01-15T22:30:01.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: .0) 401 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:30:01.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024 2024-01-15T22:30:01.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: -01-15T22:30:00.000826+0000 2024-01-15T22:30:01.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: mon.smithi099 (mon.0) 402 : cluster [WRN] /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:30:01.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 2024-01-15T22:30:01.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01 2024-01-15T22:30:01.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: -15T22:30:00.000838+0000 mon.smithi099 (mon.0) 2024-01-15T22:30:01.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 403 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:30:01.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30: 2024-01-15T22:30:01.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 00.000849+0000 mon.smithi099 (mon.0) 404 : cluster 2024-01-15T22:30:01.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: [WRN] /bin/podman: stderr self.zap_osd() 2024-01-15T22:30:01.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:01.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 2024-01-15T22:30:00.000862+0000 mon.smithi099 (mon.0) 405 2024-01-15T22:30:01.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:01.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:01.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 2024-01-15T22:30:00.000873+0000 mon.smithi099 (mon 2024-01-15T22:30:01.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: .0) 406 : cluster [WRN] /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:01.166 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22 2024-01-15T22:30:01.166 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: :30:00.000885+0000 mon.smithi099 (mon.0 2024-01-15T22:30:01.166 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: ) 407 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:30:01.166 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30 2024-01-15T22:30:01.166 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: :00.000900+0000 mon.smithi099 (mon.0) 408 2024-01-15T22:30:01.166 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: : cluster [WRN] /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:30:01.166 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:00 2024-01-15T22:30:01.166 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: .000912+0000 mon.smithi099 (mon.0) 409 : cluster 2024-01-15T22:30:01.166 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:30:01.166 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:01.166 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 2024-01-15T22:30:00.000924+0000 mon.smithi099 (mon 2024-01-15T22:30:01.166 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: .0) 410 : cluster [WRN] /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:30:01.166 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:00. 2024-01-15T22:30:01.166 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 000935+0000 mon.smithi099 (mon.0) 411 : cluster [WRN] 2024-01-15T22:30:01.166 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:30:01.166 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:01.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 2024-01-15T22:30:00.000947+0000 mon.smithi099 (mon 2024-01-15T22:30:01.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: .0) 412 : cluster [WRN] Traceback (most recent call last): 2024-01-15T22:30:01.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T 2024-01-15T22:30:01.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 22:30:00.000959+0000 mon.smithi099 (mon.0 2024-01-15T22:30:01.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: ) 413 : cluster [WRN] File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:30:01.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30: 2024-01-15T22:30:01.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 00.000970+0000 mon.smithi099 (mon.0) 414 : 2024-01-15T22:30:01.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster [WRN] main() 2024-01-15T22:30:01.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:00 2024-01-15T22:30:01.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: .000981+0000 mon.smithi099 (mon.0) 415 : cluster 2024-01-15T22:30:01.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: [WRN] File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:30:01.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:00 2024-01-15T22:30:01.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: .000993+0000 mon.smithi099 (mon.0) 416 : cluster 2024-01-15T22:30:01.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: [WRN] r = ctx.func(ctx) 2024-01-15T22:30:01.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:01.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 2024-01-15T22:30:00.001005+0000 mon.smithi099 ( 2024-01-15T22:30:01.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: mon.0) 417 : cluster [WRN] File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:30:01.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01- 2024-01-15T22:30:01.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 15T22:30:00.001016+0000 mon.smithi099 (mon 2024-01-15T22:30:01.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: .0) 418 : cluster [WRN] return func(ctx) 2024-01-15T22:30:01.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T 2024-01-15T22:30:01.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 22:30:00.001028+0000 mon.smithi099 (mon.0) 2024-01-15T22:30:01.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 419 : cluster [WRN] File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:30:01.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30: 2024-01-15T22:30:01.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 00.001039+0000 mon.smithi099 (mon.0) 420 : cluster 2024-01-15T22:30:01.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: [WRN] return func(ctx) 2024-01-15T22:30:01.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:01.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 2024-01-15T22:30:00.001051+0000 mon.smithi099 2024-01-15T22:30:01.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: (mon.0) 421 : cluster [WRN] File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:30:01.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30 2024-01-15T22:30:01.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: :00.001063+0000 mon.smithi099 (mon.0) 422 2024-01-15T22:30:01.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: : cluster [WRN] return func(ctx) 2024-01-15T22:30:01.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30: 2024-01-15T22:30:01.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 00.001074+0000 mon.smithi099 (mon.0) 423 : cluster 2024-01-15T22:30:01.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: [WRN] File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:30:01.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:01.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 2024-01-15T22:30:00.001085+0000 mon.smithi099 (mon.0) 2024-01-15T22:30:01.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 424 : cluster [WRN] return func(ctx) 2024-01-15T22:30:01.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30: 2024-01-15T22:30:01.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 00.001097+0000 mon.smithi099 (mon.0) 425 : cluster 2024-01-15T22:30:01.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: [WRN] File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:30:01.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30:01.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 2024-01-15T22:30:00.001112+0000 mon.smithi099 (mon 2024-01-15T22:30:01.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: .0) 426 : cluster [WRN] out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:30:01.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T 2024-01-15T22:30:01.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 22:30:00.001124+0000 mon.smithi099 (mon.0 2024-01-15T22:30:01.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: ) 427 : cluster [WRN] File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:30:01.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22:30 2024-01-15T22:30:01.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: :00.001138+0000 mon.smithi099 (mon.0) 428 2024-01-15T22:30:01.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: : cluster [WRN] raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:30:01.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: cluster 2024-01-15T22: 2024-01-15T22:30:01.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: 30:00.001150+0000 mon.smithi099 (mon.0) 429 2024-01-15T22:30:01.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: : cluster [WRN] RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpco61kzan:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp89e4iwv_:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:30:01.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: audit 2024-01-15T22:30:00.214449+0000 mon.smithi099 2024-01-15T22:30:01.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:00 smithi099 conmon[35292]: (mon.0) 430 : audit [DBG] from='client.? 172.21.15.99:0/625232497' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:30:01.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22:30:00.000149+0000 mon.smithi099 (mon.0) 348 : cluster [WRN] Health detail: HEALTH_WARN Failed to apply 1 service(s): osd.all-available-devices 2024-01-15T22:30:01.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22:30 2024-01-15T22:30:01.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: :00.000175+0000 mon.smithi099 (mon.0) 349 : cluster [WRN] [WRN] CEPHADM_APPLY_SPEC_FAIL: Failed to apply 1 service(s): osd.all-available-devices 2024-01-15T22:30:01.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22:30:00.000191+0000 mon.smithi099 (mon.0) 350 : cluster [WRN] osd.all-available-devices: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:30:01.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T 2024-01-15T22:30:01.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 22:30:00.000208+0000 mon.smithi099 (mon.0) 351 : cluster [WRN] Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpco61kzan:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp89e4iwv_:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:30:01.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22:30:01.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 2024-01-15T22:30: 2024-01-15T22:30:01.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 00.000220+0000 mon.smithi099 (mon.0) 352 2024-01-15T22:30:01.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: : cluster [WRN] /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:30:01.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01 2024-01-15T22:30:01.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: -15T22:30:00.000232 2024-01-15T22:30:01.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: +0000 mon.smithi099 (mon.0) 353 : cluster [WRN] /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:30:01.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 2024-01-15T22:30:01.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22:30:01.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 2024-01-15T22:30: 2024-01-15T22:30:01.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 00.000245+0000 mon.smithi099 (mon.0) 2024-01-15T22:30:01.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 354 : cluster [WRN] /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:30:01.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024- 2024-01-15T22:30:01.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 01-15T22:30:00. 2024-01-15T22:30:01.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 000259+0000 mon.smithi099 (mon.0) 355 : cluster 2024-01-15T22:30:01.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: [WRN] /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 339f1c8c-0c2c-46f4-bd37-a13d21334cb9 2024-01-15T22:30:01.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024- 2024-01-15T22:30:01.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 01-15T22:30:00. 2024-01-15T22:30:01.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 000270+0000 mon.smithi099 (mon.0) 356 : cluster 2024-01-15T22:30:01.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: [WRN] /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:30:01.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T 2024-01-15T22:30:01.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 22:30:00.000283+0000 mon.smithi099 (mon 2024-01-15T22:30:01.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: .0) 357 : cluster [WRN] /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:30:01.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024 2024-01-15T22:30:01.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: -01-15T22:30:00. 2024-01-15T22:30:01.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 000294+0000 mon.smithi099 (mon.0) 358 : cluster 2024-01-15T22:30:01.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: [WRN] /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:30:01.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22 2024-01-15T22:30:01.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: :30:00.000305+0000 mon.smithi099 (mon. 2024-01-15T22:30:01.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 0) 359 : cluster [WRN] /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:30:01.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22:30:01.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 2024-01-15T 2024-01-15T22:30:01.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 22:30:00.000317+0000 mon.smithi099 2024-01-15T22:30:01.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: (mon.0) 360 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:30:01.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22:30:01.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 2024-01 2024-01-15T22:30:01.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: -15T22:30:00.000337 2024-01-15T22:30:01.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: +0000 mon.smithi099 (mon.0) 361 : cluster [WRN] 2024-01-15T22:30:01.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: /bin/podman: stderr self.prepare() 2024-01-15T22:30:01.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15 2024-01-15T22:30:01.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: T22:30:00.000349+0000 2024-01-15T22:30:01.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: mon.smithi099 (mon.0) 362 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:01.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 2024-01-15T22:30:01.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22:30:01.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 2024-01-15T22:30:00 2024-01-15T22:30:01.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: .000361+0000 mon.smithi099 (mon.0) 363 : 2024-01-15T22:30:01.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster [WRN] /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:01.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024- 2024-01-15T22:30:01.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 01-15T22:30:00. 2024-01-15T22:30:01.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 000373+0000 mon.smithi099 (mon.0) 2024-01-15T22:30:01.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 364 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:30:01.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22:30:01.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 2024-01- 2024-01-15T22:30:01.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 15T22:30:00.000385+0000 2024-01-15T22:30:01.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: mon.smithi099 (mon.0) 365 : cluster [WRN] 2024-01-15T22:30:01.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:30:01.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15 2024-01-15T22:30:01.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: T22:30:00.000397+0000 2024-01-15T22:30:01.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: mon.smithi099 (mon.0) 366 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:30:01.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 2024-01-15T22:30:01.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22:30:01.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 2024-01-15T22:30:00 2024-01-15T22:30:01.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: .000409+0000 mon.smithi099 (mon.0) 367 : 2024-01-15T22:30:01.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster [WRN] /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:30:01.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024 2024-01-15T22:30:01.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: -01-15T22:30:00 2024-01-15T22:30:01.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: .000420+0000 mon.smithi099 (mon.0) 368 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:30:01.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22:30 2024-01-15T22:30:01.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: :00.000431+0000 mon.smithi099 (mon.0) 369 : cluster [WRN] 2024-01-15T22:30:01.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:30:01.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22 2024-01-15T22:30:01.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: :30:00.000442+0000 mon.smithi099 (mon. 2024-01-15T22:30:01.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 0) 370 : cluster [WRN] /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:30:01.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22:30:01.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 2024-01-15T22:30: 2024-01-15T22:30:01.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 00.000453+0000 mon.smithi099 (mon.0) 2024-01-15T22:30:01.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 371 : cluster [WRN] /bin/podman: stderr 2024-01-15T22:30:01.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024- 2024-01-15T22:30:01.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 01-15T22:30:00. 2024-01-15T22:30:01.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 000469+0000 mon.smithi099 (mon.0) 372 : cluster [WRN] 2024-01-15T22:30:01.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:30:01.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22 2024-01-15T22:30:01.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: :30:00.000481+0000 mon.smithi099 (mon 2024-01-15T22:30:01.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: .0) 373 : cluster [WRN] /bin/podman: stderr 2024-01-15T22:30:01.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024 2024-01-15T22:30:01.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: -01-15T22:30:00. 2024-01-15T22:30:01.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 000493+0000 mon.smithi099 (mon.0) 374 : cluster [WRN] /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:30:01.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22 2024-01-15T22:30:01.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: :30:00.000504+0000 mon.smithi099 (mon 2024-01-15T22:30:01.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: .0) 375 : cluster [WRN] /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:30:01.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024 2024-01-15T22:30:01.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: -01-15T22:30:00. 2024-01-15T22:30:01.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 000516+0000 mon.smithi099 (mon.0) 376 : cluster [WRN] 2024-01-15T22:30:01.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:30:01.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22 2024-01-15T22:30:01.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: :30:00.000527+0000 mon.smithi099 (mon. 2024-01-15T22:30:01.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 0) 377 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:30:01.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024 2024-01-15T22:30:01.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: -01-15T22:30:00 2024-01-15T22:30:01.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: .000539+0000 mon.smithi099 (mon.0) 378 : 2024-01-15T22:30:01.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster [WRN] /bin/podman: stderr self.main(self.argv) 2024-01-15T22:30:01.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15 2024-01-15T22:30:01.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: T22:30:00.000550+0000 2024-01-15T22:30:01.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: mon.smithi099 (mon.0) 379 : cluster [WRN] 2024-01-15T22:30:01.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:30:01.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024- 2024-01-15T22:30:01.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 01-15T22:30:00. 2024-01-15T22:30:01.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 000561+0000 mon.smithi099 (mon.0) 380 : cluster 2024-01-15T22:30:01.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: [WRN] /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:30:01.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T 2024-01-15T22:30:01.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 22:30:00.000573+0000 mon.smithi099 2024-01-15T22:30:01.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: (mon.0) 381 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:30:01.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22:30:01.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 2024-01-15T22:30: 2024-01-15T22:30:01.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 00.000586+0000 mon.smithi099 (mon.0) 382 : 2024-01-15T22:30:01.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster [WRN] /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:30:01.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01 2024-01-15T22:30:01.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: -15T22:30:00.000597 2024-01-15T22:30:01.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: +0000 mon.smithi099 (mon.0) 383 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:30:01.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 2024-01-15T22:30:01.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22:30:01.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 2024-01-15T22:30:00.000609 2024-01-15T22:30:01.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: +0000 mon.smithi099 (mon.0) 384 : cluster [WRN] 2024-01-15T22:30:01.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:30:01.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22:30:01.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 2024-01-15T22:30: 2024-01-15T22:30:01.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 00.000620+0000 mon.smithi099 (mon.0) 385 2024-01-15T22:30:01.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:30:01.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01- 2024-01-15T22:30:01.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 15T22:30:00.000632+0000 2024-01-15T22:30:01.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: mon.smithi099 (mon.0) 386 : cluster [WRN] /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:30:01.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 2024-01-15T22:30:01.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22 2024-01-15T22:30:01.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: :30:00.000642+0000 mon.smithi099 (mon 2024-01-15T22:30:01.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: .0) 387 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:30:01.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22:30:01.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 2024-01-15T22:30: 2024-01-15T22:30:01.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 00.000652+0000 mon.smithi099 (mon.0) 388 2024-01-15T22:30:01.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: : cluster [WRN] /bin/podman: stderr instance.main() 2024-01-15T22:30:01.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024- 2024-01-15T22:30:01.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 01-15T22:30:00.000664 2024-01-15T22:30:01.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: +0000 mon.smithi099 (mon.0) 389 : cluster [WRN] 2024-01-15T22:30:01.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:01.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22:30:00. 2024-01-15T22:30:01.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 000679+0000 mon.smithi099 (mon.0) 390 : cluster [WRN] 2024-01-15T22:30:01.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:01.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01 2024-01-15T22:30:01.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: -15T22:30:00.000691 2024-01-15T22:30:01.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: +0000 mon.smithi099 (mon.0) 391 : cluster [WRN] 2024-01-15T22:30:01.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:30:01.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T 2024-01-15T22:30:01.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 22:30:00.000703+0000 mon.smithi099 2024-01-15T22:30:01.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: (mon.0) 392 : cluster [WRN] /bin/podman: stderr self._execute(plan) 2024-01-15T22:30:01.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22:30:01.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 2024-01-15T22:30: 2024-01-15T22:30:01.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 00.000714+0000 mon.smithi099 (mon.0) 393 : cluster 2024-01-15T22:30:01.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:30:01.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15 2024-01-15T22:30:01.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: T22:30:00.000726+0000 2024-01-15T22:30:01.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: mon.smithi099 (mon.0) 394 : cluster [WRN] 2024-01-15T22:30:01.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:30:01.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024- 2024-01-15T22:30:01.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 01-15T22:30:00. 2024-01-15T22:30:01.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 000737+0000 mon.smithi099 (mon.0) 395 : cluster 2024-01-15T22:30:01.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:01.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15 2024-01-15T22:30:01.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: T22:30:00.000748+0000 mon.smithi099 ( 2024-01-15T22:30:01.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: mon.0) 396 : cluster [WRN] /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:01.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22:30:01.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 2024-01-15T22:30:00 2024-01-15T22:30:01.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: .000760+0000 mon.smithi099 (mon.0) 397 : 2024-01-15T22:30:01.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:30:01.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01- 2024-01-15T22:30:01.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 15T22:30:00.000771 2024-01-15T22:30:01.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: +0000 mon.smithi099 (mon.0) 398 : cluster [WRN] /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:30:01.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 2024-01-15T22:30:01.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22:30:01.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 2024-01-15T22:30:00.000792 2024-01-15T22:30:01.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: +0000 mon.smithi099 (mon.0) 399 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:30:01.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22:30:01.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 2024-01-15T22: 2024-01-15T22:30:01.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 30:00.000803+0000 mon.smithi099 (mon 2024-01-15T22:30:01.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: .0) 400 : cluster [WRN] /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:30:01.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22:30:01.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 2024-01-15T22:30: 2024-01-15T22:30:01.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 00.000815+0000 mon.smithi099 (mon.0) 401 2024-01-15T22:30:01.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:30:01.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15 2024-01-15T22:30:01.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: T22:30:00.000826+0000 mon.smithi099 2024-01-15T22:30:01.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: (mon.0) 402 : cluster [WRN] /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:30:01.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22:30:00.000838+0000 mon.smithi099 2024-01-15T22:30:01.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: (mon.0) 403 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:30:01.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22 2024-01-15T22:30:01.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: :30:00.000849+0000 mon.smithi099 (mon 2024-01-15T22:30:01.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: .0) 404 : cluster [WRN] /bin/podman: stderr self.zap_osd() 2024-01-15T22:30:01.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024 2024-01-15T22:30:01.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: -01-15T22:30:00 2024-01-15T22:30:01.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: .000862+0000 mon.smithi099 (mon.0) 405 : 2024-01-15T22:30:01.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:01.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01- 2024-01-15T22:30:01.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 15T22:30:00.000873+0000 2024-01-15T22:30:01.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: mon.smithi099 (mon.0) 406 : cluster [WRN] /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:01.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 2024-01-15T22:30:01.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01 2024-01-15T22:30:01.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: -15T22:30:00.000885+0000 mon.smithi099 2024-01-15T22:30:01.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: (mon.0) 407 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:30:01.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22:30:01.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 2024-01-15T22:30 2024-01-15T22:30:01.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: :00.000900+0000 mon.smithi099 (mon 2024-01-15T22:30:01.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: .0) 408 : cluster [WRN] /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:30:01.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22:30:01.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 2024-01- 2024-01-15T22:30:01.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 15T22:30:00.000912+0000 2024-01-15T22:30:01.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: mon.smithi099 (mon.0) 409 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:30:01.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22:30:01.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 2024-01-15T22:30 2024-01-15T22:30:01.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: :00.000924+0000 mon.smithi099 (mon.0) 2024-01-15T22:30:01.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 410 : cluster [WRN] /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:30:01.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024- 2024-01-15T22:30:01.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 01-15T22:30:00.000935+0000 2024-01-15T22:30:01.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: mon.smithi099 (mon.0) 411 : cluster [WRN] /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:30:01.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22:30:01.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 2024-01- 2024-01-15T22:30:01.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 15T22:30:00.000947+0000 2024-01-15T22:30:01.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: mon.smithi099 (mon.0) 412 : cluster [WRN] Traceback (most recent call last): 2024-01-15T22:30:01.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024- 2024-01-15T22:30:01.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 01-15T22:30:00.000959+0000 mon.smithi099 2024-01-15T22:30:01.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: (mon.0) 413 : cluster [WRN] File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:30:01.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22:30:01.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 2024-01-15 2024-01-15T22:30:01.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: T22:30:00.000970+0000 mon.smithi099 ( 2024-01-15T22:30:01.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: mon.0) 414 : cluster [WRN] main() 2024-01-15T22:30:01.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22:30:01.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 2024-01-15T22:30:00 2024-01-15T22:30:01.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: .000981+0000 mon.smithi099 (mon.0) 415 : cluster 2024-01-15T22:30:01.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: [WRN] File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:30:01.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22 2024-01-15T22:30:01.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: :30:00.000993+0000 mon.smithi099 (mon. 2024-01-15T22:30:01.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 0) 416 : cluster [WRN] r = ctx.func(ctx) 2024-01-15T22:30:01.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22:30:01.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 2024-01-15T22:30 2024-01-15T22:30:01.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: :00.001005+0000 mon.smithi099 (mon.0 2024-01-15T22:30:01.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: ) 417 : cluster [WRN] File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:30:01.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024- 2024-01-15T22:30:01.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 01-15T22:30:00. 2024-01-15T22:30:01.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 001016+0000 mon.smithi099 (mon.0) 418 : cluster [WRN] 2024-01-15T22:30:01.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:30:01.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T 2024-01-15T22:30:01.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 22:30:00.001028+0000 mon.smithi099 2024-01-15T22:30:01.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: (mon.0) 419 : cluster [WRN] File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:30:01.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22:30:01.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 2024-01-15T22:30:00.001039+0000 2024-01-15T22:30:01.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: mon.smithi099 (mon.0) 420 : cluster [WRN] return func(ctx) 2024-01-15T22:30:01.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024- 2024-01-15T22:30:01.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 01-15T22:30:00.001051+0000 2024-01-15T22:30:01.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: mon.smithi099 (mon.0) 421 : cluster [WRN] File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:30:01.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22:30:01.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 2024-01-15T22: 2024-01-15T22:30:01.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 30:00.001063+0000 mon.smithi099 (mon. 2024-01-15T22:30:01.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 0) 422 : cluster [WRN] return func(ctx) 2024-01-15T22:30:01.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024 2024-01-15T22:30:01.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: -01-15T22:30:00. 2024-01-15T22:30:01.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 001074+0000 mon.smithi099 (mon.0) 2024-01-15T22:30:01.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 423 : cluster [WRN] File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:30:01.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22:30:01.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 2024-01-15T22:30: 2024-01-15T22:30:01.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 00.001085+0000 mon.smithi099 (mon.0 2024-01-15T22:30:01.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: ) 424 : cluster [WRN] return func(ctx) 2024-01-15T22:30:01.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024- 2024-01-15T22:30:01.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 01-15T22:30:00. 2024-01-15T22:30:01.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 001097+0000 mon.smithi099 (mon.0) 425 : cluster 2024-01-15T22:30:01.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: [WRN] File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:30:01.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024 2024-01-15T22:30:01.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: -01-15T22:30:00 2024-01-15T22:30:01.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: .001112+0000 mon.smithi099 (mon.0) 426 : cluster 2024-01-15T22:30:01.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: [WRN] out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:30:01.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15 2024-01-15T22:30:01.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: T22:30:00.001124+0000 2024-01-15T22:30:01.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: mon.smithi099 (mon.0) 427 : cluster [WRN] File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:30:01.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01-15T22:30:01.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 2024-01-15T22:30: 2024-01-15T22:30:01.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 00.001138+0000 mon.smithi099 (mon.0) 428 2024-01-15T22:30:01.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: : cluster [WRN] raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:30:01.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: cluster 2024-01- 2024-01-15T22:30:01.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 15T22:30:00.001150+0000 2024-01-15T22:30:01.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: mon.smithi099 (mon.0) 429 : cluster [WRN] RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpco61kzan:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp89e4iwv_:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:30:01.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: 2024-01-15T22:30:01.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: audit 2024 2024-01-15T22:30:01.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: -01-15T22:30:00.214449+0000 mon.smithi099 2024-01-15T22:30:01.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:00 smithi162 conmon[39755]: (mon.0) 430 : audit [DBG] from='client.? 172.21.15.99:0/625232497' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:30:01.802 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:30:02.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:01 smithi162 conmon[39755]: cluster 2024-01-15T22:30:01.753004+0000 mgr.smithi099.cmjeyn (mgr.14184) 109 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:02.303 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:01 smithi099 conmon[35292]: cluster 2024-01-15T22:30:01.753004+0000 mgr.smithi099.cmjeyn (mgr.14184) 109 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:03.594 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:30:03.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:03 smithi099 conmon[35292]: audit 2024-01-15T22:30:03.593530+0000 mon.smithi099 (mon.0) 431 : audit [DBG] from='client.? 172.21.15.99:0/923811445' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:30:03.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:03 smithi162 conmon[39755]: audit 2024-01-15T22:30:03.593530+0000 mon.smithi099 (mon.0) 431 : audit [DBG] from='client.? 172.21.15.99:0/923811445' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:30:04.142 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:30:04.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:04 smithi099 conmon[35292]: cluster 2024-01-15T22:30:03.753359+0000 mgr.smithi099.cmjeyn (mgr.14184) 110 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:04.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:04 smithi162 conmon[39755]: cluster 2024-01-15T22:30:03.753359+0000 mgr.smithi099.cmjeyn (mgr.14184) 110 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:05.143 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:30:06.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:05 smithi162 conmon[39755]: audit 2024-01-15T22:30:05.741418+0000 mon.smithi099 (mon.0) 432 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-15T22:30:06.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:05 smithi162 conmon[39755]: cluster 2024-01-15T22:30:05.753747+0000 mgr.smithi099.cmjeyn (mgr.14184) 111 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:06.318 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:05 smithi099 conmon[35292]: audit 2024-01-15T22:30:05.741418+0000 2024-01-15T22:30:06.318 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:05 smithi099 conmon[35292]: mon.smithi099 (mon.0) 432 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-15T22:30:06.318 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:05 smithi099 conmon[35292]: cluster 2024-01-15T22:30:05.753747+0000 mgr.smithi099.cmjeyn (mgr.14184) 111 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:06.873 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:30:07.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:06 smithi099 conmon[35292]: audit 2024-01-15T22:30:06.872931+0000 mon.smithi099 (mon.0) 433 : audit [DBG] from='client.? 172.21.15.99:0/2527579479' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:30:07.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:06 smithi162 conmon[39755]: audit 2024-01-15T22:30:06.872931+0000 mon.smithi099 (mon.0) 433 2024-01-15T22:30:07.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:06 smithi162 conmon[39755]: : audit [DBG] from='client.? 172.21.15.99:0/2527579479' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:30:07.562 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:30:08.563 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:30:08.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:08 smithi099 conmon[35292]: audit 2024-01-15T22:30:07. 2024-01-15T22:30:08.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:08 smithi099 conmon[35292]: 552461+0000 mon.smithi099 (mon.0) 434 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:30:08.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:08 smithi099 conmon[35292]: cluster 2024-01-15T22:30:07.754311+0000 mgr.smithi099.cmjeyn (mgr.14184) 112 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:08.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:08 smithi099 conmon[35292]: audit 2024-01-15T22:30:07.845253+0000 mon.smithi099 (mon.0) 435 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:30:08.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:08 smithi099 conmon[35292]: audit 2024-01-15T22: 2024-01-15T22:30:08.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:08 smithi099 conmon[35292]: 30:07.845934+0000 mon.smithi099 (mon.0) 436 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-15T22:30:08.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:08 smithi099 conmon[35292]: audit 2024-01-15T 2024-01-15T22:30:08.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:08 smithi099 conmon[35292]: 22:30:07.850219+0000 mon.smithi099 (mon.0) 437 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:30:08.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:08 smithi099 conmon[35292]: audit 2024-01- 2024-01-15T22:30:08.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:08 smithi099 conmon[35292]: 15T22:30:07.853966+0000 mon.smithi099 (mon.0) 438 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-15T22:30:08.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:08 smithi099 conmon[35292]: audit 2024 2024-01-15T22:30:08.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:08 smithi099 conmon[35292]: -01-15T22:30:07.857016+0000 mon.smithi099 (mon.0) 439 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-15T22:30:08.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:08 smithi099 conmon[35292]: audit 2024-01-15T22:30:07.857904+0000 mon.smithi099 (mon.0) 440 : audit 2024-01-15T22:30:08.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:08 smithi099 conmon[35292]: [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-15T22:30:08.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:08 smithi099 conmon[35292]: audit 2024-01-15T22:30 2024-01-15T22:30:08.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:08 smithi099 conmon[35292]: :07.858063+0000 mon.smithi099 (mon.0) 441 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:30:08.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:08 smithi099 conmon[35292]: audit 2024 2024-01-15T22:30:08.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:08 smithi099 conmon[35292]: -01-15T22:30:07.858490+0000 mon.smithi099 (mon.0) 442 : audit 2024-01-15T22:30:08.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:08 smithi099 conmon[35292]: [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:30:08.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:08 smithi162 conmon[39755]: audit 2024-01-15T22:30:07.552461+0000 mon.smithi099 (mon.0) 434 2024-01-15T22:30:08.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:08 smithi162 conmon[39755]: : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:30:08.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:08 smithi162 conmon[39755]: cluster 2024-01-15T22:30:07.754311+0000 mgr.smithi099.cmjeyn (mgr.14184) 112 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:08.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:08 smithi162 conmon[39755]: audit 2024-01-15T22:30:07.845253+0000 mon.smithi099 (mon.0) 435 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:30:08.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:08 smithi162 conmon[39755]: audit 2024-01-15T22:30:07.845934+0000 mon.smithi099 (mon.0) 436 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-15T22:30:08.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:08 smithi162 conmon[39755]: audit 2024-01-15T22:30:07.850219+0000 mon.smithi099 (mon.0) 437 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:30:08.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:08 smithi162 conmon[39755]: audit 2024-01-15T22:30:07.853966+0000 mon.smithi099 (mon.0) 438 : audit [DBG] 2024-01-15T22:30:08.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:08 smithi162 conmon[39755]: from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-15T22:30:08.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:08 smithi162 conmon[39755]: audit 2024-01-15T22:30:07.857016+0000 mon.smithi099 (mon.0) 439 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-15T22:30:08.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:08 smithi162 conmon[39755]: audit 2024-01-15T22:30:08.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:08 smithi162 conmon[39755]: 2024-01-15T22:30:07.857904+0000 mon.smithi099 (mon.0) 440 : audit 2024-01-15T22:30:08.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:08 smithi162 conmon[39755]: [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-15T22:30:08.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:08 smithi162 conmon[39755]: audit 2024-01-15T22:30:08.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:08 smithi162 conmon[39755]: 2024-01-15T22:30:07.858063+0000 mon.smithi099 ( 2024-01-15T22:30:08.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:08 smithi162 conmon[39755]: mon.0) 441 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:30:08.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:08 smithi162 conmon[39755]: audit 2024-01-15T 2024-01-15T22:30:08.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:08 smithi162 conmon[39755]: 22:30:07.858490+0000 mon.smithi099 (mon.0 2024-01-15T22:30:08.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:08 smithi162 conmon[39755]: ) 442 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:30:09.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:09 smithi099 conmon[35292]: cluster 2024-01-15T22:30:07.850551+0000 mgr.smithi099.cmjeyn (mgr.14184) 113 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:09.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:09 smithi099 conmon[35292]: cluster 2024-01-15T22:30:08.550682+0000 mon.smithi099 (mon.0) 443 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-15T22:30:09.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:09 smithi099 conmon[35292]: cluster 2024-01-15T22:30:08.550709+0000 mon.smithi099 (mon.0) 444 : cluster [INF] Cluster is now healthy 2024-01-15T22:30:09.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:09 smithi162 conmon[39755]: cluster 2024-01-15T22:30:07.850551+0000 mgr.smithi099.cmjeyn (mgr.14184) 113 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:09.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:09 smithi162 conmon[39755]: cluster 2024-01-15T22:30:08.550682+0000 mon.smithi099 (mon.0) 443 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-15T22:30:09.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:09 smithi162 conmon[39755]: cluster 2024-01-15T22:30:08.550709+0000 mon.smithi099 (mon.0) 444 : cluster [INF] Cluster is now healthy 2024-01-15T22:30:10.693 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:30:11.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:10 smithi099 conmon[35292]: cluster 2024-01-15T22:30:09.850817+0000 mgr.smithi099.cmjeyn (mgr.14184) 114 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:11.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:10 smithi099 conmon[35292]: audit 2024-01-15T22:30:10.449866+0000 mon.smithi162 (mon.1) 9 : audit [INF] from='client.? 172.21.15.162:0/2276006261' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71b58a0a-2973-4712-a541-e366be1c2ae1"}]: dispatch 2024-01-15T22:30:11.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:10 smithi099 conmon[35292]: audit 2024-01-15T22:30:10.450249+0000 mon.smithi099 (mon.0) 445 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71b58a0a-2973-4712-a541-e366be1c2ae1"}]: dispatch 2024-01-15T22:30:11.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:10 smithi099 conmon[35292]: audit 2024-01-15T22:30:10.455202+0000 mon.smithi099 (mon.0) 446 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71b58a0a-2973-4712-a541-e366be1c2ae1"}]': finished 2024-01-15T22:30:11.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:10 smithi099 conmon[35292]: cluster 2024-01-15T22:30:10.455292+0000 mon.smithi099 (mon.0) 447 : cluster [DBG] osdmap e18: 1 total, 0 up, 1 in 2024-01-15T22:30:11.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:10 smithi099 conmon[35292]: audit 2024-01-15T22:30:10.455431+0000 mon.smithi099 (mon.0) 448 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-15T22:30:11.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:10 smithi099 conmon[35292]: audit 2024-01-15T22:30:10.692693+0000 mon.smithi099 (mon.0) 449 : audit [DBG] from='client.? 172.21.15.99:0/323520819' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:30:11.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:10 smithi099 conmon[35292]: audit 2024-01-15T22:30:10.746852+0000 mon.smithi162 (mon.1) 10 : audit [INF] from='client.? 172.21.15.162:0/1498198115' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:30:11.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:10 smithi099 conmon[35292]: audit 2024-01-15T22:30:10.747269+0000 mon.smithi099 (mon.0) 450 : audit 2024-01-15T22:30:11.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:10 smithi099 conmon[35292]: [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:30:11.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:10 smithi099 conmon[35292]: audit 2024-01-15T22:30:10.751536+0000 mon.smithi099 (mon.0) 451 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-15T22:30:11.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:10 smithi099 conmon[35292]: cluster 2024-01-15T22:30:10.751586+0000 mon.smithi099 (mon.0) 452 : cluster [DBG] osdmap e19: 0 total, 0 up, 0 in 2024-01-15T22:30:11.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:10 smithi162 conmon[39755]: cluster 2024-01-15T22:30:09.850817+0000 mgr.smithi099.cmjeyn (mgr.14184 2024-01-15T22:30:11.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:10 smithi162 conmon[39755]: ) 114 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:11.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:10 smithi162 conmon[39755]: audit 2024-01-15T22:30:10.449866+0000 mon.smithi162 (mon.1) 9 : audit [INF] from='client.? 172.21.15.162:0/2276006261' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71b58a0a-2973-4712-a541-e366be1c2ae1"}]: dispatch 2024-01-15T22:30:11.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:10 smithi162 conmon[39755]: audit 2024-01-15T22:30:10.450249+0000 mon.smithi099 (mon.0) 445 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71b58a0a-2973-4712-a541-e366be1c2ae1"}]: dispatch 2024-01-15T22:30:11.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:10 smithi162 conmon[39755]: audit 2024-01-15T22:30:10.455202+0000 mon.smithi099 (mon.0) 446 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71b58a0a-2973-4712-a541-e366be1c2ae1"}]': finished 2024-01-15T22:30:11.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:10 smithi162 conmon[39755]: cluster 2024-01-15T22:30:10.455292+0000 mon.smithi099 (mon.0) 447 : cluster [DBG] osdmap e18: 1 total, 0 up, 1 in 2024-01-15T22:30:11.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:10 smithi162 conmon[39755]: audit 2024-01-15T22:30:10.455431+0000 mon.smithi099 (mon.0) 448 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-15T22:30:11.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:10 smithi162 conmon[39755]: audit 2024-01-15T22:30:10.692693+0000 mon.smithi099 (mon.0) 449 : audit [DBG] from='client.? 172.21.15.99:0/323520819' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:30:11.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:10 smithi162 conmon[39755]: audit 2024-01-15T22:30:10.746852+0000 mon.smithi162 (mon.1) 10 : audit [INF] from='client.? 172.21.15.162:0/1498198115' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:30:11.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:10 smithi162 conmon[39755]: audit 2024-01-15T22:30:10.747269+0000 mon.smithi099 (mon.0) 450 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:30:11.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:10 smithi162 conmon[39755]: audit 2024-01-15T22:30:10.751536+0000 mon.smithi099 (mon.0) 451 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-15T22:30:11.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:10 smithi162 conmon[39755]: cluster 2024-01-15T22:30:10.751586+0000 mon.smithi099 (mon.0) 452 : cluster [DBG] osdmap e19: 0 total, 0 up, 0 in 2024-01-15T22:30:11.381 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":18,"num_osds":1,"num_up_osds":0,"osd_up_since":0,"num_in_osds":1,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:30:11.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: audit 2024-01-15T22:30:11.052770+0000 mon.smithi099 (mon.0) 453 : audit [INF] from='client.? 172.21.15.99:0/2587760728' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "405be3c1-177a-4c31-9dcf-5796644513d2"}]: dispatch 2024-01-15T22:30:11.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: 2024-01-15T22:30:11.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: audit 2024-01-15T22:30:11.056392+0000 mon.smithi099 (mon.0) 454 : audit [INF] from='client.? 172.21.15.99:0/2587760728' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "405be3c1-177a-4c31-9dcf-5796644513d2"}]': finished 2024-01-15T22:30:11.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: cluster 2024-01-15T22:30:11.056453+0000 mon.smithi099 (mon.0) 455 : cluster [DBG] osdmap e20: 1 total, 0 up, 1 in 2024-01-15T22:30:11.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: audit 2024-01-15T22:30:11.056552 2024-01-15T22:30:11.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: +0000 mon.smithi099 (mon.0) 456 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-15T22:30:11.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: audit 2024-01-15T22:30:11.349087+0000 mon.smithi099 (mon 2024-01-15T22:30:11.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: .0) 457 : audit [INF] from='client.? 172.21.15.99:0/3454898517' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:30:11.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: audit 2024-01-15T22:30:11.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: 2024-01-15T22:30:11.354304+0000 mon.smithi099 (mon.0) 458 : audit [INF] from='client.? 172.21.15.99:0/3454898517' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-15T22:30:11.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: cluster 2024-01-15T22:30:11.354370+0000 2024-01-15T22:30:11.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: mon.smithi099 (mon.0) 459 : cluster [DBG] osdmap e21: 0 total, 0 up, 0 in 2024-01-15T22:30:11.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: cephadm 2024-01-15T22:30:11. 2024-01-15T22:30:11.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: 489909+0000 mgr.smithi099.cmjeyn (mgr.14184) 115 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi162', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-15T22:30:11.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:30:11.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:30:11.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: return f(*arg) 2024-01-15T22:30:11.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:30:11.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:30:11.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:30:11.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: code, '\n'.join(err))) 2024-01-15T22:30:11.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:30:11.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpginbvwnd:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpsxihsc1d:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:30:11.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:30:11.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:30:11.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:30:11.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 71b58a0a-2973-4712-a541-e366be1c2ae1 2024-01-15T22:30:11.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:30:11.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:30:11.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:30:11.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:30:11.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:30:11.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr self.prepare() 2024-01-15T22:30:11.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:11.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:11.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:30:11.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:30:11.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:30:11.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:30:11.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:30:11.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:30:11.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:30:11.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:30:11.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:30:11.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:30:11.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:30:11.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:30:11.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:30:11.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:30:11.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:30:11.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:30:11.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:30:11.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:30:11.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:30:11.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:30:11.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:30:11.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:30:11.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:30:11.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:30:11.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:30:11.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:11.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:11.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:30:11.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:30:11.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:30:12.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:30:12.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:12.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:12.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:30:12.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:30:12.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:30:12.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:30:12.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:30:12.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:30:12.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:30:12.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:30:12.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:12.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:12.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:30:12.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:30:12.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:30:12.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:30:12.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:30:12.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:30:12.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:30:12.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: main() 2024-01-15T22:30:12.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:30:12.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: r = ctx.func(ctx) 2024-01-15T22:30:12.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:30:12.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:30:12.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:30:12.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:30:12.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:30:12.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:30:12.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d2 2024-01-15T22:30:12.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: 9dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:30:12.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:30:12.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:30:12.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:30:12.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:30:12.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:30:12.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:11 smithi099 conmon[35292]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpginbvwnd:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpsxihsc1d:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:30:12.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: audit 2024-01-15T22:30:11.052770+0000 mon.smithi099 (mon.0) 453 : audit [INF] from='client.? 172.21.15.99:0/2587760728' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "405be3c1-177a-4c31-9dcf-5796644513d2"}]: dispatch 2024-01-15T22:30:12.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: 2024-01-15T22:30:12.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: audit 2024-01-15T22:30:11.056392+0000 mon.smithi099 (mon.0) 454 : audit [INF] from='client.? 172.21.15.99:0/2587760728' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "405be3c1-177a-4c31-9dcf-5796644513d2"}]': finished 2024-01-15T22:30:12.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: cluster 2024-01-15T22:30:11.056453+0000 mon.smithi099 (mon.0) 455 : cluster [DBG] osdmap e20: 1 total, 0 up, 1 in 2024-01-15T22:30:12.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: audit 2024-01-15T22:30:11.056552+0000 mon.smithi099 (mon.0) 456 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-15T22:30:12.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: audit 2024-01-15T22:30:11.349087+0000 mon.smithi099 (mon.0) 457 : audit [INF] from='client.? 172.21.15.99:0/3454898517' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:30:12.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: audit 2024-01-15T22:30:11.354304+0000 mon.smithi099 (mon.0) 458 : audit [INF] from='client.? 172.21.15.99:0/3454898517' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-15T22:30:12.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: cluster 2024-01-15T22:30:11.354370+0000 mon.smithi099 (mon.0) 459 : cluster [DBG] osdmap e21: 0 total, 0 up, 0 in 2024-01-15T22:30:12.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: cephadm 2024-01-15T22:30:11.489909+0000 mgr.smithi099.cmjeyn (mgr.14184) 115 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi162', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-15T22:30:12.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:30:12.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:30:12.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: return f(*arg) 2024-01-15T22:30:12.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:30:12.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:30:12.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:30:12.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: code, '\n'.join(err))) 2024-01-15T22:30:12.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:30:12.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpginbvwnd:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpsxihsc1d:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:30:12.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:30:12.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:30:12.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:30:12.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 71b58a0a-2973-4712-a541-e366be1c2ae1 2024-01-15T22:30:12.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:30:12.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:30:12.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:30:12.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:30:12.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:30:12.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr self.prepare() 2024-01-15T22:30:12.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:12.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:12.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:30:12.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:30:12.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:30:12.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:30:12.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:30:12.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:30:12.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:30:12.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:30:12.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:30:12.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:30:12.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:30:12.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:30:12.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:30:12.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:30:12.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:30:12.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:30:12.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:30:12.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:30:12.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:30:12.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:30:12.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:30:12.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:30:12.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:30:12.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:30:12.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:30:12.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:12.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:12.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:30:12.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:30:12.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:30:12.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:30:12.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:12.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:12.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:30:12.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:30:12.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:30:12.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:30:12.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:30:12.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:30:12.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:30:12.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:30:12.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:12.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:12.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:30:12.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:30:12.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py" 2024-01-15T22:30:12.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: , line 88, in find_associated_devices 2024-01-15T22:30:12.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:30:12.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:30:12.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:30:12.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:30:12.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: main() 2024-01-15T22:30:12.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:30:12.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: r = ctx.func(ctx) 2024-01-15T22:30:12.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:30:12.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:30:12.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:30:12.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:30:12.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:30:12.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:30:12.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:30:12.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:30:12.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:30:12.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:30:12.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:30:12.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:30:12.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:11 smithi162 conmon[39755]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpginbvwnd:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpsxihsc1d:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:30:12.382 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:30:13.137 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: cluster 2024-01-15T22:30:11.851073+0000 mgr.smithi099.cmjeyn (mgr.14184) 116 : cluster 2024-01-15T22:30:13.138 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:13.138 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: cephadm 2024-01-15T22:30:11.988804+0000 mgr.smithi099.cmjeyn (mgr.14184) 117 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi162', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-15T22:30:13.138 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:30:13.138 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:30:13.138 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: return f(*arg) 2024-01-15T22:30:13.138 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:30:13.138 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:30:13.139 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:30:13.139 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: code, '\n'.join(err))) 2024-01-15T22:30:13.139 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi099/config 2024-01-15T22:30:13.139 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp_c2j1si5:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpyivrx27n:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:30:13.139 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:30:13.139 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:30:13.139 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:30:13.139 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 405be3c1-177a-4c31-9dcf-5796644513d2 2024-01-15T22:30:13.139 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:30:13.139 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:30:13.139 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:30:13.139 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:30:13.139 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:30:13.139 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr self.prepare() 2024-01-15T22:30:13.140 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:13.140 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:13.140 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:30:13.140 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:30:13.140 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:30:13.140 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:30:13.140 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:30:13.140 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:30:13.140 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:30:13.140 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:30:13.140 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:30:13.140 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:30:13.140 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:30:13.140 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:30:13.141 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:30:13.141 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:30:13.141 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:30:13.141 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:30:13.141 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:30:13.141 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:30:13.141 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:30:13.141 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:30:13.141 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:30:13.141 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:30:13.141 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:30:13.141 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:30:13.141 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:30:13.141 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:13.141 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:13.142 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:30:13.142 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:30:13.142 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:30:13.142 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:30:13.142 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:13.142 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:13.142 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:30:13.142 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:30:13.142 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:30:13.142 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:30:13.142 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:30:13.142 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:30:13.142 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:30:13.142 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:30:13.143 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:13.143 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:13.143 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:30:13.143 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:30:13.143 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:30:13.143 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:30:13.143 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:30:13.143 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:30:13.143 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:30:13.143 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: main() 2024-01-15T22:30:13.143 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:30:13.143 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: r = ctx.func(ctx) 2024-01-15T22:30:13.144 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:30:13.144 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:30:13.144 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:30:13.144 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:30:13.144 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:30:13.144 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: retu 2024-01-15T22:30:13.144 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: rn func(ctx) 2024-01-15T22:30:13.144 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:30:13.144 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:30:13.144 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:30:13.144 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:30:13.144 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:30:13.144 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:30:13.145 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp_c2j1si5:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpyivrx27n:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:30:13.145 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: cephadm 2024-01-15T22:30:11.989918+0000 mgr.smithi099.cmjeyn (mgr.14184) 118 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-15T22:30:13.145 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: service_id: all-available-devices 2024-01-15T22:30:13.145 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: service_name: osd.all-available-devices 2024-01-15T22:30:13.145 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: placement: 2024-01-15T22:30:13.145 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: host_pattern: '*' 2024-01-15T22:30:13.145 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: spec: 2024-01-15T22:30:13.145 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: data_devices: 2024-01-15T22:30:13.145 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: all: true 2024-01-15T22:30:13.145 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: filter_logic: AND 2024-01-15T22:30:13.145 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: objectstore: bluestore 2024-01-15T22:30:13.145 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:30:13.145 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpginbvwnd:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpsxihsc1d:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:30:13.145 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:30:13.145 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:30:13.146 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:30:13.146 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 71b58a0a-2973-4712-a541-e366be1c2ae1 2024-01-15T22:30:13.146 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:30:13.146 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:30:13.146 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:30:13.146 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:30:13.146 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:30:13.146 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr self.prepare() 2024-01-15T22:30:13.146 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:13.146 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:13.146 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:30:13.146 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:30:13.146 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:30:13.146 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:30:13.147 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:30:13.147 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:30:13.147 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:30:13.147 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:30:13.147 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:30:13.147 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:30:13.147 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:30:13.147 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:30:13.147 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:30:13.147 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:30:13.147 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:30:13.147 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:30:13.147 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:30:13.147 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:30:13.147 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:30:13.148 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:30:13.148 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:30:13.148 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:30:13.148 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:30:13.148 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:30:13.148 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:30:13.148 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:13.148 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:13.148 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:30:13.148 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:30:13.148 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:30:13.148 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:30:13.148 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:13.148 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:13.149 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:30:13.149 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:30:13.149 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:30:13.149 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:30:13.149 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:30:13.149 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:30:13.149 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:30:13.149 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:30:13.149 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:13.149 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:13.149 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stder 2024-01-15T22:30:13.149 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: r File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:30:13.149 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:30:13.149 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:30:13.150 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:30:13.150 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:30:13.150 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:30:13.150 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:30:13.150 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: main() 2024-01-15T22:30:13.150 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:30:13.150 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: r = ctx.func(ctx) 2024-01-15T22:30:13.150 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:30:13.150 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:30:13.150 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:30:13.150 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:30:13.150 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:30:13.150 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:30:13.150 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:30:13.150 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:30:13.151 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:30:13.151 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:30:13.151 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:30:13.151 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:30:13.151 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpginbvwnd:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpsxihsc1d:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:30:13.151 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:30:13.151 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-15T22:30:13.151 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: if self._apply_service(spec): 2024-01-15T22:30:13.151 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-15T22:30:13.151 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-15T22:30:13.151 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-15T22:30:13.151 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-15T22:30:13.151 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-15T22:30:13.151 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-15T22:30:13.151 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-15T22:30:13.152 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-15T22:30:13.152 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-15T22:30:13.152 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: raise self._value 2024-01-15T22:30:13.152 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-15T22:30:13.152 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: result = (True, func(*args, **kwds)) 2024-01-15T22:30:13.152 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-15T22:30:13.152 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: return list(map(*args)) 2024-01-15T22:30:13.152 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:30:13.152 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: return f(*arg) 2024-01-15T22:30:13.152 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:30:13.152 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:30:13.152 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:30:13.152 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: code, '\n'.join(err))) 2024-01-15T22:30:13.152 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:30:13.152 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpginbvwnd:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpsxihsc1d:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:30:13.153 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:30:13.153 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:30:13.153 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:30:13.153 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 71b58a0a-2973-4712-a541-e366be1c2ae1 2024-01-15T22:30:13.153 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:30:13.153 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:30:13.153 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:30:13.153 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:30:13.153 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:30:13.153 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr self.prepare() 2024-01-15T22:30:13.153 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:13.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:13.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:30:13.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:30:13.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:30:13.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:30:13.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:30:13.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:30:13.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:30:13.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:30:13.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:30:13.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:30:13.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:30:13.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:30:13.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:30:13.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:30:13.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bi 2024-01-15T22:30:13.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: n/podman: stderr self.main(self.argv) 2024-01-15T22:30:13.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:30:13.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:30:13.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:30:13.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:30:13.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:30:13.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:30:13.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:30:13.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:30:13.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:30:13.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:30:13.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:13.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:13.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:30:13.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:30:13.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:30:13.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:30:13.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:13.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:13.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:30:13.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:30:13.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:30:13.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:30:13.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:30:13.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:30:13.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:30:13.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:30:13.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:13.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:13.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:30:13.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:30:13.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:30:13.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:30:13.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:30:13.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:30:13.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:30:13.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: main() 2024-01-15T22:30:13.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:30:13.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: r = ctx.func(ctx) 2024-01-15T22:30:13.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:30:13.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:30:13.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:30:13.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:30:13.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:30:13.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:30:13.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:30:13.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:30:13.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:30:13.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:30:13.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:30:13.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:30:13.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpginbvwnd:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpsxihsc1d:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:30:13.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: cluster 2024-01-15T22:30:11.990408+0000 mgr.smithi099.cmjeyn (mgr.14184) 119 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:13.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:12 smithi099 conmon[35292]: cluster 2024-01-15T22:30:12.351104+0000 mon.smithi099 (mon.0) 460 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-15T22:30:13.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: cluster 2024-01-15T22:30:11.851073+0000 mgr.smithi099.cmjeyn (mgr.14184) 116 : cluster [DBG] 2024-01-15T22:30:13.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:13.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: cephadm 2024-01-15T22:30:11.988804+0000 mgr.smithi099.cmjeyn (mgr.14184) 117 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi162', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-15T22:30:13.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:30:13.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:30:13.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: return f(*arg) 2024-01-15T22:30:13.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:30:13.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:30:13.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:30:13.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: code, '\n'.join(err))) 2024-01-15T22:30:13.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi099/config 2024-01-15T22:30:13.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp_c2j1si5:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpyivrx27n:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:30:13.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:30:13.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:30:13.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:30:13.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 405be3c1-177a-4c31-9dcf-5796644513d2 2024-01-15T22:30:13.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:30:13.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:30:13.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:30:13.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:30:13.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:30:13.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr self.prepare() 2024-01-15T22:30:13.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:13.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:13.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:30:13.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:30:13.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:30:13.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:30:13.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:30:13.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:30:13.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:30:13.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:30:13.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:30:13.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:30:13.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:30:13.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:30:13.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:30:13.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:30:13.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:30:13.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:30:13.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:30:13.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:30:13.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:30:13.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:30:13.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:30:13.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:30:13.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:30:13.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:30:13.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:30:13.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:13.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:13.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:30:13.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:30:13.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:30:13.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:30:13.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:13.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:13.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:30:13.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:30:13.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:30:13.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:30:13.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:30:13.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:30:13.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:30:13.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:30:13.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:13.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:13.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:30:13.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:30:13.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:30:13.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:30:13.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:30:13.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:30:13.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:30:13.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: main() 2024-01-15T22:30:13.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:30:13.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: r = ctx.func(ctx) 2024-01-15T22:30:13.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:30:13.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:30:13.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:30:13.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:30:13.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:30:13.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: return func 2024-01-15T22:30:13.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: (ctx) 2024-01-15T22:30:13.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:30:13.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:30:13.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:30:13.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:30:13.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:30:13.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:30:13.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp_c2j1si5:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpyivrx27n:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:30:13.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: cephadm 2024-01-15T22:30:11.989918+0000 mgr.smithi099.cmjeyn (mgr.14184) 118 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-15T22:30:13.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: service_id: all-available-devices 2024-01-15T22:30:13.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: service_name: osd.all-available-devices 2024-01-15T22:30:13.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: placement: 2024-01-15T22:30:13.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: host_pattern: '*' 2024-01-15T22:30:13.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: spec: 2024-01-15T22:30:13.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: data_devices: 2024-01-15T22:30:13.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: all: true 2024-01-15T22:30:13.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: filter_logic: AND 2024-01-15T22:30:13.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: objectstore: bluestore 2024-01-15T22:30:13.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:30:13.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpginbvwnd:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpsxihsc1d:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:30:13.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:30:13.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:30:13.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:30:13.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 71b58a0a-2973-4712-a541-e366be1c2ae1 2024-01-15T22:30:13.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:30:13.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:30:13.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:30:13.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:30:13.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:30:13.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr self.prepare() 2024-01-15T22:30:13.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:13.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:13.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:30:13.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:30:13.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:30:13.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:30:13.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:30:13.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:30:13.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:30:13.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:30:13.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:30:13.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:30:13.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:30:13.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:30:13.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:30:13.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:30:13.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:30:13.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:30:13.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:30:13.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:30:13.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:30:13.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:30:13.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:30:13.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:30:13.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:30:13.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:30:13.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:30:13.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:13.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:13.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:30:13.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:30:13.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:30:13.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:30:13.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:13.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:13.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:30:13.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:30:13.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:30:13.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:30:13.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:30:13.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:30:13.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:30:13.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:30:13.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:13.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:13.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr Fil 2024-01-15T22:30:13.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: e "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:30:13.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:30:13.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:30:13.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:30:13.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:30:13.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:30:13.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:30:13.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: main() 2024-01-15T22:30:13.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:30:13.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: r = ctx.func(ctx) 2024-01-15T22:30:13.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:30:13.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:30:13.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:30:13.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:30:13.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:30:13.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:30:13.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:30:13.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:30:13.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:30:13.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:30:13.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:30:13.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:30:13.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpginbvwnd:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpsxihsc1d:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:30:13.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:30:13.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-15T22:30:13.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: if self._apply_service(spec): 2024-01-15T22:30:13.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-15T22:30:13.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-15T22:30:13.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-15T22:30:13.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-15T22:30:13.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-15T22:30:13.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-15T22:30:13.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-15T22:30:13.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-15T22:30:13.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-15T22:30:13.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: raise self._value 2024-01-15T22:30:13.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-15T22:30:13.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: result = (True, func(*args, **kwds)) 2024-01-15T22:30:13.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-15T22:30:13.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: return list(map(*args)) 2024-01-15T22:30:13.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:30:13.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: return f(*arg) 2024-01-15T22:30:13.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:30:13.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:30:13.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:30:13.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: code, '\n'.join(err))) 2024-01-15T22:30:13.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:30:13.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpginbvwnd:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpsxihsc1d:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:30:13.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:30:13.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:30:13.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:30:13.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 71b58a0a-2973-4712-a541-e366be1c2ae1 2024-01-15T22:30:13.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:30:13.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:30:13.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:30:13.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:30:13.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:30:13.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr self.prepare() 2024-01-15T22:30:13.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:13.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:13.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:30:13.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:30:13.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:30:13.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:30:13.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:30:13.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:30:13.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:30:13.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:30:13.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:30:13.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:30:13.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:30:13.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:30:13.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:30:13.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:30:13.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podma 2024-01-15T22:30:13.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: n: stderr self.main(self.argv) 2024-01-15T22:30:13.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:30:13.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:30:13.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:30:13.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:30:13.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:30:13.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:30:13.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:30:13.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:30:13.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:30:13.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:30:13.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:13.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:13.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:30:13.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:30:13.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:30:13.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:30:13.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:13.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:13.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:30:13.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:30:13.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:30:13.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:30:13.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:30:13.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:30:13.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:30:13.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:30:13.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:30:13.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:30:13.219 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:30:13.219 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:30:13.219 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:30:13.219 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:30:13.219 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:30:13.219 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:30:13.219 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:30:13.219 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: main() 2024-01-15T22:30:13.219 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:30:13.219 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: r = ctx.func(ctx) 2024-01-15T22:30:13.219 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:30:13.219 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:30:13.219 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:30:13.219 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:30:13.219 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:30:13.220 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:30:13.220 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:30:13.220 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:30:13.220 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:30:13.220 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:30:13.220 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:30:13.220 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:30:13.220 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpginbvwnd:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpsxihsc1d:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:30:13.220 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: cluster 2024-01-15T22:30:11.990408+0000 mgr.smithi099.cmjeyn (mgr.14184) 119 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:13.220 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:12 smithi162 conmon[39755]: cluster 2024-01-15T22:30:12.351104+0000 mon.smithi099 (mon.0) 460 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-15T22:30:14.153 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:30:14.739 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:30:15.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:15 smithi099 conmon[35292]: cluster 2024-01-15T22:30:13.990759+0000 mgr.smithi099.cmjeyn (mgr.14184) 120 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:15.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:15 smithi099 conmon[35292]: audit 2024-01-15T22:30:14.153603+0000 mon.smithi099 (mon.0) 461 : audit [DBG] from='client.? 172.21.15.99:0/1738400909' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:30:15.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:15 smithi162 conmon[39755]: cluster 2024-01-15T22:30:13.990759+0000 mgr.smithi099.cmjeyn (mgr.14184) 120 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:15.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:15 smithi162 conmon[39755]: audit 2024-01-15T22:30:14.153603+0000 mon.smithi099 (mon.0) 461 : audit [DBG] from='client.? 172.21.15.99:0/1738400909' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:30:15.740 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:30:17.216 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:17 smithi099 conmon[35292]: cluster 2024-01-15T22:30:15.991127+0000 mgr.smithi099.cmjeyn (mgr.14184) 121 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:17.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:17 smithi162 conmon[39755]: cluster 2024-01-15T22:30:15.991127+0000 mgr.smithi099.cmjeyn (mgr.14184) 121 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:17.520 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:30:18.089 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:30:18.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:18 smithi099 conmon[35292]: audit 2024-01-15T22:30:17.520591+0000 mon.smithi162 (mon.1 2024-01-15T22:30:18.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:18 smithi099 conmon[35292]: ) 11 : audit [DBG] from='client.? 172.21.15.99:0/1283016274' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:30:18.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:18 smithi162 conmon[39755]: audit 2024-01-15T22:30:17.520591+0000 mon.smithi162 (mon.1) 11 : audit [DBG] from='client.? 172.21.15.99:0/1283016274' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:30:19.091 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:30:19.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:19 smithi099 conmon[35292]: cluster 2024-01-15T22:30:17.991616+0000 mgr.smithi099.cmjeyn (mgr.14184) 122 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:19.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:19 smithi162 conmon[39755]: cluster 2024-01-15T22:30:17.991616+0000 mgr.smithi099.cmjeyn (mgr.14184) 122 : cluster 2024-01-15T22:30:19.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:19 smithi162 conmon[39755]: [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:20.858 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:30:21.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:20 smithi099 conmon[35292]: cluster 2024-01-15T22:30:19.992141+0000 mgr.smithi099.cmjeyn (mgr.14184) 123 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:21.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:20 smithi099 conmon[35292]: audit 2024-01-15T22:30:20.857206+0000 mon.smithi099 (mon.0) 462 : audit [DBG] from='client.? 172.21.15.99:0/3158297212' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:30:21.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:20 smithi162 conmon[39755]: cluster 2024-01-15T22:30:19.992141+0000 mgr.smithi099.cmjeyn (mgr.14184) 123 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:21.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:20 smithi162 conmon[39755]: 2024-01-15T22:30:21.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:20 smithi162 conmon[39755]: audit 2024-01-15T22:30:20.857206+0000 mon.smithi099 (mon.0) 462 : audit [DBG] from='client.? 172.21.15.99:0/3158297212' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:30:21.522 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:30:22.524 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:30:23.331 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:23 smithi099 conmon[35292]: cluster 2024-01-15T22:30:21.992585+0000 mgr.smithi099.cmjeyn (mgr.14184) 124 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:23.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:23 smithi162 conmon[39755]: cluster 2024-01-15T22:30:21.992585+0000 mgr.smithi099.cmjeyn (mgr.14184) 124 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:24.326 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:30:24.947 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:30:25.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:25 smithi099 conmon[35292]: cluster 2024-01-15T22:30:23.992959+0000 mgr.smithi099.cmjeyn (mgr.14184) 125 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:25.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:25 smithi099 conmon[35292]: audit 2024-01-15T22:30:24.326119+0000 mon.smithi099 (mon.0) 463 : audit [DBG] from='client.? 172.21.15.99:0/3996295263' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:30:25.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:25 smithi162 conmon[39755]: cluster 2024-01-15T22:30:23.992959+0000 mgr.smithi099.cmjeyn (mgr.14184) 125 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:25.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:25 smithi162 conmon[39755]: audit 2024-01-15T22:30:24.326119+0000 mon.smithi099 (mon.0) 463 : audit [DBG] from='client.? 172.21.15.99:0/3996295263' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:30:25.948 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:30:27.332 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:27 smithi099 conmon[35292]: cluster 2024-01-15T22:30:25.993530+0000 mgr.smithi099.cmjeyn (mgr.14184) 126 : 2024-01-15T22:30:27.332 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:27 smithi099 conmon[35292]: cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:27.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:27 smithi162 conmon[39755]: cluster 2024-01-15T22:30:25.993530+0000 mgr.smithi099.cmjeyn (mgr.14184) 126 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:27.676 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:30:28.228 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:30:28.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:28 smithi099 conmon[35292]: audit 2024-01-15T22:30:27.677115+0000 mon.smithi099 (mon.0) 464 : audit [DBG] from='client.? 172.21.15.99:0/1255330684' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:30:28.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:28 smithi162 conmon[39755]: audit 2024-01-15T22:30:27.677115+0000 mon.smithi099 (mon.0) 464 : audit [DBG] from='client.? 172.21.15.99:0/1255330684' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:30:29.229 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:30:29.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:29 smithi099 conmon[35292]: cluster 2024-01-15T22:30:27.993990+0000 mgr.smithi099.cmjeyn (mgr.14184) 127 : cluster 2024-01-15T22:30:29.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:29 smithi099 conmon[35292]: [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:29.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:29 smithi162 conmon[39755]: cluster 2024-01-15T22:30:27.993990+0000 mgr.smithi099.cmjeyn (mgr.14184) 127 : cluster 2024-01-15T22:30:29.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:29 smithi162 conmon[39755]: [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:30.993 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:30:31.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:30 smithi162 conmon[39755]: cluster 2024-01-15T22:30:29.994540+0000 mgr.smithi099.cmjeyn (mgr.14184) 128 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:31.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:30 smithi099 conmon[35292]: cluster 2024-01-15T22:30:29.994540+0000 mgr.smithi099.cmjeyn (mgr.14184) 128 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:31.528 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:30:32.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:31 smithi162 conmon[39755]: audit 2024-01-15T22:30:30.993013+0000 mon.smithi099 (mon.0) 465 : audit [DBG] from='client.? 172.21.15.99:0/3690460718' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:30:32.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:31 smithi099 conmon[35292]: audit 2024-01-15T22:30:30.993013+0000 mon.smithi099 (mon.0) 465 : audit [DBG] 2024-01-15T22:30:32.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:31 smithi099 conmon[35292]: from='client.? 172.21.15.99:0/3690460718' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:30:32.528 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:30:33.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:32 smithi162 conmon[39755]: cluster 2024-01-15T22:30:31.994972+0000 mgr.smithi099.cmjeyn (mgr.14184) 129 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:33.355 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:32 smithi099 conmon[35292]: cluster 2024-01-15T22:30:31.994972+0000 mgr.smithi099.cmjeyn (mgr.14184) 129 : cluster [DBG] 2024-01-15T22:30:33.355 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:32 smithi099 conmon[35292]: pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:34.325 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:30:34.903 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:30:35.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:35 smithi099 conmon[35292]: cluster 2024-01-15T22:30:33.995567+0000 mgr.smithi099.cmjeyn (mgr.14184) 130 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:35.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:35 smithi099 conmon[35292]: audit 2024-01-15T22:30:34.324623+0000 mon.smithi099 (mon.0) 466 : audit [DBG] from='client.? 172.21.15.99:0/1797829525' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:30:35.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:35 smithi162 conmon[39755]: cluster 2024-01-15T22:30:33.995567+0000 mgr.smithi099.cmjeyn (mgr.14184) 130 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:35.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:35 smithi162 conmon[39755]: audit 2024-01-15T22:30:34.324623+0000 mon.smithi099 (mon.0) 466 : 2024-01-15T22:30:35.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:35 smithi162 conmon[39755]: audit [DBG] from='client.? 172.21.15.99:0/1797829525' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:30:35.905 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:30:37.329 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:37 smithi099 conmon[35292]: cluster 2024-01-15T22:30:35.995989+0000 mgr.smithi099.cmjeyn (mgr.14184) 131 : cluster 2024-01-15T22:30:37.329 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:37 smithi099 conmon[35292]: [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:37.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:37 smithi162 conmon[39755]: cluster 2024-01-15T22:30:35.995989+0000 mgr.smithi099.cmjeyn (mgr.14184) 131 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:37.692 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:30:38.235 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:30:38.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:38 smithi099 conmon[35292]: audit 2024-01-15T22:30:37.692324+0000 mon.smithi099 (mon.0) 467 : audit [DBG] from='client.? 172.21.15.99:0/2103479525' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:30:38.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:38 smithi162 conmon[39755]: audit 2024-01-15T22:30:37.692324+0000 mon.smithi099 (mon.0) 467 : audit [DBG] from='client.? 172.21.15.99:0/2103479525' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:30:39.237 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:30:39.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:39 smithi099 conmon[35292]: cluster 2024-01-15T22:30:37.996320+0000 mgr.smithi099.cmjeyn (mgr.14184) 132 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:39.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:39 smithi162 conmon[39755]: cluster 2024-01-15T22:30:37.996320+0000 mgr.smithi099.cmjeyn (mgr.14184) 132 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:41.018 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:30:41.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:40 smithi162 conmon[39755]: cluster 2024-01-15T22:30:39.996653+0000 mgr.smithi099.cmjeyn (mgr.14184) 133 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:41.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:40 smithi099 conmon[35292]: cluster 2024-01-15T22:30:39.996653+0000 mgr.smithi099.cmjeyn (mgr.14184) 133 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:41.609 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:30:42.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:41 smithi162 conmon[39755]: audit 2024-01-15T22:30:41.018040+0000 mon.smithi099 (mon.0) 468 : audit [DBG] from='client.? 172.21.15.99:0/321518124' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:30:42.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:41 smithi099 conmon[35292]: audit 2024-01-15T22:30:41.018040+0000 mon.smithi099 (mon 2024-01-15T22:30:42.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:41 smithi099 conmon[35292]: .0) 468 : audit [DBG] from='client.? 172.21.15.99:0/321518124' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:30:42.610 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:30:43.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:42 smithi162 conmon[39755]: cluster 2024-01-15T22:30:41.997046+0000 mgr.smithi099.cmjeyn (mgr.14184) 134 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:43.197 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:42 smithi099 conmon[35292]: cluster 2024-01-15T22:30:41.997046+0000 mgr.smithi099.cmjeyn (mgr.14184) 134 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:44.300 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:30:44.827 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:30:45.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:45 smithi099 conmon[35292]: cluster 2024-01-15T22:30:43.997392+0000 mgr.smithi099.cmjeyn (mgr.14184) 135 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:45.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:45 smithi099 conmon[35292]: audit 2024- 2024-01-15T22:30:45.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:45 smithi099 conmon[35292]: 01-15T22:30:44.300284+0000 mon.smithi099 (mon.0) 469 : audit [DBG] from='client.? 172.21.15.99:0/1277448024' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:30:45.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:45 smithi162 conmon[39755]: cluster 2024-01-15T22:30:43.997392+0000 mgr.smithi099.cmjeyn (mgr.14184) 135 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:45.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:45 smithi162 conmon[39755]: audit 2024-01-15T22:30:44.300284+0000 mon.smithi099 (mon.0) 469 : audit [DBG] from='client.? 172.21.15.99:0/1277448024' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:30:45.828 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:30:47.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:47 smithi099 conmon[35292]: cluster 2024-01-15T22:30:45.997784+0000 mgr.smithi099.cmjeyn (mgr.14184) 136 : 2024-01-15T22:30:47.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:47 smithi099 conmon[35292]: cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:47.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:47 smithi162 conmon[39755]: cluster 2024-01-15T22:30:45.997784+0000 mgr.smithi099.cmjeyn (mgr.14184) 136 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:47.573 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:30:48.117 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:30:48.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:48 smithi099 conmon[35292]: audit 2024-01-15T22:30:47.573581+0000 mon.smithi099 (mon.0) 470 : audit [DBG] from='client.? 172.21.15.99:0/1876808088' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:30:48.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:48 smithi162 conmon[39755]: audit 2024-01-15T22:30:47.573581+0000 mon.smithi099 (mon.0) 470 : audit [DBG] from='client.? 172.21.15.99:0/1876808088' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:30:49.118 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:30:49.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:49 smithi099 conmon[35292]: cluster 2024-01-15T22:30:47.998404+0000 mgr.smithi099.cmjeyn (mgr.14184) 2024-01-15T22:30:49.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:49 smithi099 conmon[35292]: 137 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:49.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:49 smithi162 conmon[39755]: cluster 2024-01-15T22:30:47.998404+0000 mgr.smithi099.cmjeyn (mgr.14184) 137 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:50.908 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:30:51.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:50 smithi162 conmon[39755]: cluster 2024-01-15T22:30:49.998776+0000 mgr.smithi099.cmjeyn (mgr.14184) 138 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:51.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:50 smithi162 conmon[39755]: audit 2024-01-15T22:30:50.908590+0000 mon.smithi099 (mon.0) 471 : audit [DBG] from='client.? 172.21.15.99:0/3628241975' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:30:51.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:50 smithi099 conmon[35292]: cluster 2024-01-15T22:30:49.998776+0000 2024-01-15T22:30:51.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:50 smithi099 conmon[35292]: mgr.smithi099.cmjeyn (mgr.14184) 138 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:51.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:50 smithi099 conmon[35292]: audit 2024-01-15T22:30:50.908590+0000 mon.smithi099 (mon.0) 471 : audit [DBG] from='client.? 172.21.15.99:0/3628241975' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:30:51.491 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:30:52.492 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:30:53.316 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:53 smithi099 conmon[35292]: cluster 2024-01-15T22:30:51.999121+0000 mgr.smithi099.cmjeyn (mgr.14184) 139 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:53.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:53 smithi162 conmon[39755]: cluster 2024-01-15T22:30:51.999121+0000 mgr.smithi099.cmjeyn (mgr.14184) 139 : cluster 2024-01-15T22:30:53.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:53 smithi162 conmon[39755]: [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:54.276 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:30:54.815 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:30:55.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:55 smithi099 conmon[35292]: cluster 2024-01-15T22:30:53.999505+0000 mgr.smithi099.cmjeyn (mgr.14184) 140 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:55.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:55 smithi099 conmon[35292]: audit 2024-01-15T22:30:54.276886+0000 mon.smithi099 (mon.0) 472 : audit [DBG] from='client.? 172.21.15.99:0/2232544973' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:30:55.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:55 smithi162 conmon[39755]: cluster 2024-01-15T22:30:53.999505+0000 mgr.smithi099.cmjeyn (mgr.14184) 140 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:55.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:55 smithi162 conmon[39755]: audit 2024-01-15T22:30:54. 2024-01-15T22:30:55.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:55 smithi162 conmon[39755]: 276886+0000 mon.smithi099 (mon.0) 472 : audit [DBG] from='client.? 172.21.15.99:0/2232544973' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:30:55.816 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:30:57.262 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:57 smithi099 conmon[35292]: cluster 2024-01-15T22:30:55.999922+0000 mgr.smithi099.cmjeyn (mgr.14184 2024-01-15T22:30:57.262 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:57 smithi099 conmon[35292]: ) 141 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:57.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:57 smithi162 conmon[39755]: cluster 2024-01-15T22:30:55.999922+0000 mgr.smithi099.cmjeyn (mgr.14184) 141 : cluster [DBG] 2024-01-15T22:30:57.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:57 smithi162 conmon[39755]: pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:57.584 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:30:58.173 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:30:58.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:58 smithi099 conmon[35292]: audit 2024-01-15T22:30:57.584327+0000 mon.smithi099 (mon.0 2024-01-15T22:30:58.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:58 smithi099 conmon[35292]: ) 473 : audit [DBG] from='client.? 172.21.15.99:0/1542686326' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:30:58.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:58 smithi162 conmon[39755]: audit 2024-01-15T22:30:57.584327+0000 mon.smithi099 (mon.0) 473 : audit [DBG] 2024-01-15T22:30:58.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:58 smithi162 conmon[39755]: from='client.? 172.21.15.99:0/1542686326' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:30:59.174 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:30:59.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:59 smithi099 conmon[35292]: cluster 2024-01-15T22:30:58.000293+0000 mgr.smithi099.cmjeyn (mgr.14184) 142 : cluster [DBG] 2024-01-15T22:30:59.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:30:59 smithi099 conmon[35292]: pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:30:59.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:30:59 smithi162 conmon[39755]: cluster 2024-01-15T22:30:58.000293+0000 mgr.smithi099.cmjeyn (mgr.14184) 142 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:01.003 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:31:01.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:00 smithi162 conmon[39755]: cluster 2024-01-15T22:31:00.001008+0000 mgr.smithi099.cmjeyn (mgr.14184) 143 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:01.206 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:00 smithi099 conmon[35292]: cluster 2024-01-15T22:31:00.001008+0000 mgr.smithi099.cmjeyn (mgr.14184) 143 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:01.589 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:31:02.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:01 smithi162 conmon[39755]: audit 2024-01-15T22:31:01.002455+0000 mon.smithi099 (mon.0) 474 : audit [DBG] from='client.? 172.21.15.99:0/3162356185' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:31:02.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:01 smithi099 conmon[35292]: audit 2024-01-15T22:31:01.002455+0000 mon.smithi099 (mon.0) 474 : audit [DBG] from='client.? 172.21.15.99:0/3162356185' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:31:02.590 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:31:03.233 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:02 smithi099 conmon[35292]: cluster 2024-01-15T22:31:02.001564+0000 mgr.smithi099.cmjeyn (mgr.14184) 144 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:03.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:02 smithi162 conmon[39755]: cluster 2024-01-15T22:31:02.001564+0000 mgr.smithi099.cmjeyn (mgr.14184) 144 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:04.357 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:31:04.938 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:31:05.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:05 smithi099 conmon[35292]: cluster 2024-01-15T22:31:04.002227+0000 mgr.smithi099.cmjeyn (mgr.14184) 145 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:05.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:05 smithi099 conmon[35292]: audit 2024-01-15T22:31:04.357524+0000 mon.smithi099 (mon.0) 475 : audit [DBG] from='client.? 172.21.15.99:0/3603044116' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:31:05.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:05 smithi162 conmon[39755]: cluster 2024-01-15T22:31:04.002227+0000 mgr.smithi099.cmjeyn (mgr.14184) 145 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:05.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:05 smithi162 conmon[39755]: audit 2024-01-15T22:31:04.357524+0000 mon.smithi099 (mon.0) 475 : audit [DBG] from='client.? 172.21.15.99:0/3603044116' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:31:05.939 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:31:07.314 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:07 smithi099 conmon[35292]: cluster 2024-01-15T22:31:06.002756+0000 mgr.smithi099.cmjeyn (mgr.14184) 146 : cluster 2024-01-15T22:31:07.314 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:07 smithi099 conmon[35292]: [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:07.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:07 smithi162 conmon[39755]: cluster 2024-01-15T22:31:06.002756+0000 mgr.smithi099.cmjeyn (mgr.14184) 146 : cluster 2024-01-15T22:31:07.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:07 smithi162 conmon[39755]: [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:07.665 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:31:08.212 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:31:08.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:08 smithi099 conmon[35292]: audit 2024-01-15T22:31:07.665017+0000 mon.smithi099 (mon.0) 476 : audit [DBG] from='client.? 172.21.15.99:0/1727318419' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:31:08.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:08 smithi162 conmon[39755]: audit 2024-01-15T22:31:07.665017+0000 mon.smithi099 (mon.0) 476 : audit [DBG] from='client.? 172.21.15.99:0/1727318419' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:31:09.213 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:31:09.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:09 smithi099 conmon[35292]: cluster 2024-01-15T22:31:08.003167+0000 mgr.smithi099.cmjeyn (mgr.14184) 147 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:09.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:09 smithi162 conmon[39755]: cluster 2024-01-15T22:31:08.003167+0000 mgr.smithi099.cmjeyn (mgr.14184) 147 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:11.001 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:31:11.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:10 smithi162 conmon[39755]: cluster 2024-01-15T22:31:10.003774+0000 mgr.smithi099.cmjeyn (mgr.14184) 148 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:11.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:10 smithi099 conmon[35292]: cluster 2024-01-15T22:31:10.003774+0000 mgr.smithi099.cmjeyn (mgr.14184) 148 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:11.586 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:31:12.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:11 smithi162 conmon[39755]: audit 2024-01-15T22:31:11.001044+0000 mon.smithi099 (mon.0) 477 : audit [DBG] from='client.? 172.21.15.99:0/2930316932' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:31:12.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:11 smithi099 conmon[35292]: audit 2024-01-15T22:31:11.001044+0000 mon.smithi099 (mon.0) 477 2024-01-15T22:31:12.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:11 smithi099 conmon[35292]: : audit [DBG] from='client.? 172.21.15.99:0/2930316932' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:31:12.587 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:31:13.349 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:12 smithi099 conmon[35292]: audit 2024-01-15T22:31:11.991709+0000 mon.smithi099 ( 2024-01-15T22:31:13.349 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:12 smithi099 conmon[35292]: mon.0) 478 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-15T22:31:13.349 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:12 smithi099 conmon[35292]: cluster 2024-01-15T22:31:12.004142+0000 mgr.smithi099.cmjeyn (mgr.14184) 149 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:13.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:12 smithi162 conmon[39755]: audit 2024-01-15T22:31:11.991709+0000 mon.smithi099 (mon.0) 478 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-15T22:31:13.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:12 smithi162 conmon[39755]: cluster 2024-01-15T22:31:12.004142+0000 mgr.smithi099.cmjeyn (mgr.14184) 149 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:14.314 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:31:14.906 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:31:14.929 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:14 smithi162 conmon[39755]: audit 2024-01-15T22:31:13.801155+0000 mon.smithi099 (mon 2024-01-15T22:31:14.929 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:14 smithi162 conmon[39755]: .0) 479 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:31:14.929 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:14 smithi162 conmon[39755]: audit 2024-01-15T22:31:14.105208+0000 mon.smithi099 (mon.0) 480 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:31:14.929 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:14 smithi162 conmon[39755]: audit 2024-01-15T22:31:14.105838+0000 mon.smithi099 (mon.0) 481 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-15T22:31:14.929 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:14 smithi162 conmon[39755]: audit 2024-01-15T22:31:14.109552+0000 mon.smithi099 (mon.0) 482 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:31:14.929 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:14 smithi162 conmon[39755]: audit 2024-01-15T22:31:14.113640+0000 mon.smithi099 (mon.0) 483 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-15T22:31:14.929 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:14 smithi162 conmon[39755]: audit 2024-01-15T22:31:14.116342+0000 mon.smithi099 (mon.0) 484 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-15T22:31:14.929 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:14 smithi162 conmon[39755]: audit 2024-01-15T22:31:14.117209+0000 mon.smithi099 (mon.0) 485 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-15T22:31:14.929 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:14 smithi162 conmon[39755]: audit 2024-01-15T22:31:14.117399+0000 mon.smithi099 (mon.0) 486 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:31:14.929 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:14 smithi162 conmon[39755]: audit 2024-01-15T22:31:14.117905+0000 mon.smithi099 (mon.0) 487 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:31:14.930 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:14 smithi162 conmon[39755]: audit 2024-01-15T22:31:14.315059+0000 mon.smithi099 (mon.0) 488 : audit [DBG] from='client.? 172.21.15.99:0/3307393280' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:31:15.102 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:14 smithi099 conmon[35292]: audit 2024-01-15T22:31:13.801155+0000 mon.smithi099 (mon. 2024-01-15T22:31:15.103 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:14 smithi099 conmon[35292]: 0) 479 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:31:15.103 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:14 smithi099 conmon[35292]: audit 2024-01-15T22:31:14.105208+0000 mon.smithi099 (mon.0) 480 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:31:15.103 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:14 smithi099 conmon[35292]: audit 2024-01-15T22:31:14.105838+0000 mon.smithi099 (mon.0) 481 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-15T22:31:15.103 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:14 smithi099 conmon[35292]: audit 2024-01-15T22:31:14.109552+0000 mon.smithi099 (mon.0) 482 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:31:15.103 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:14 smithi099 conmon[35292]: audit 2024-01-15T22:31:14.113640+0000 mon.smithi099 (mon.0) 483 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-15T22:31:15.103 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:14 smithi099 conmon[35292]: audit 2024-01-15T22:31:14.116342+0000 mon.smithi099 (mon.0) 484 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-15T22:31:15.103 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:14 smithi099 conmon[35292]: audit 2024-01-15T22:31:14.117209+0000 mon.smithi099 (mon.0) 485 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-15T22:31:15.103 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:14 smithi099 conmon[35292]: audit 2024-01-15T22:31:14.117399+0000 mon.smithi099 (mon.0) 486 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:31:15.103 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:14 smithi099 conmon[35292]: audit 2024-01-15T22:31:14.117905+0000 mon.smithi099 (mon.0) 487 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:31:15.103 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:14 smithi099 conmon[35292]: audit 2024-01-15T22:31:14.315059+0000 mon.smithi099 (mon.0) 488 : audit [DBG] from='client.? 172.21.15.99:0/3307393280' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:31:15.908 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:31:16.073 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:15 smithi099 conmon[35292]: cluster 2024-01-15T22:31:14.004553+0000 mgr.smithi099.cmjeyn (mgr.14184) 150 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:16.074 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:15 smithi099 conmon[35292]: cluster 2024-01-15T22:31:14.110349+0000 mgr.smithi099.cmjeyn 2024-01-15T22:31:16.074 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:15 smithi099 conmon[35292]: (mgr.14184) 151 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:16.074 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:15 smithi099 conmon[35292]: cluster 2024-01-15T22:31:14.799534+0000 mon.smithi099 (mon.0) 489 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-15T22:31:16.074 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:15 smithi099 conmon[35292]: cluster 2024-01-15T22:31:14.799563+0000 mon.smithi099 (mon.0) 490 : cluster [INF] Cluster is now healthy 2024-01-15T22:31:16.074 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:15 smithi099 conmon[35292]: audit 2024-01-15T22:31:15.357697+0000 mon.smithi099 (mon.0) 491 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:31:16.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:15 smithi162 conmon[39755]: cluster 2024-01-15T22:31:14.004553+0000 mgr.smithi099.cmjeyn (mgr.14184) 150 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:16.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:15 smithi162 conmon[39755]: cluster 2024-01-15T22:31:14.110349+0000 mgr.smithi099.cmjeyn (mgr.14184) 151 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:16.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:15 smithi162 conmon[39755]: cluster 2024-01-15T22:31:14.799534+0000 mon.smithi099 (mon.0) 489 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-15T22:31:16.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:15 smithi162 conmon[39755]: cluster 2024-01-15T22:31:14.799563+0000 mon.smithi099 (mon.0) 490 : cluster [INF] Cluster is now healthy 2024-01-15T22:31:16.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:15 smithi162 conmon[39755]: audit 2024-01-15T22:31:15.357697+0000 mon.smithi099 (mon.0) 491 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:31:17.071 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:16 smithi099 conmon[35292]: audit 2024-01-15T22:31:16.602948+0000 mon.smithi162 (mon.1) 12 : audit [INF] from='client.? 172.21.15.162:0/3507895337' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4cec51eb-d537-428a-96ea-bfcf39581618"}]: dispatch 2024-01-15T22:31:17.071 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:16 smithi099 conmon[35292]: audit 2024-01-15T22:31:16.603412+0000 mon.smithi099 (mon.0) 492 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4cec51eb-d537-428a-96ea-bfcf39581618"}]: dispatch 2024-01-15T22:31:17.071 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:16 smithi099 conmon[35292]: audit 2024-01-15T22:31:16.606681+0000 mon.smithi099 (mon.0) 493 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4cec51eb-d537-428a-96ea-bfcf39581618"}]': finished 2024-01-15T22:31:17.071 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:16 smithi099 conmon[35292]: cluster 2024-01-15T22:31:16.606732+0000 mon.smithi099 (mon.0) 494 : cluster [DBG] osdmap e22: 1 total, 0 up, 1 in 2024-01-15T22:31:17.071 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:16 smithi099 conmon[35292]: audit 2024-01-15T22:31:16.606871+0000 mon.smithi099 (mon.0) 495 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-15T22:31:17.162 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:16 smithi162 conmon[39755]: audit 2024-01-15T22:31:16.602948+0000 mon.smithi162 (mon.1) 12 : audit [INF] from='client.? 172.21.15.162:0/3507895337' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4cec51eb-d537-428a-96ea-bfcf39581618"}]: dispatch 2024-01-15T22:31:17.162 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:16 smithi162 conmon[39755]: 2024-01-15T22:31:17.162 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:16 smithi162 conmon[39755]: audit 2024-01-15T22:31:16.603412+0000 mon.smithi099 (mon.0) 492 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4cec51eb-d537-428a-96ea-bfcf39581618"}]: dispatch 2024-01-15T22:31:17.162 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:16 smithi162 conmon[39755]: audit 2024-01-15T22:31:16.606681+0000 mon.smithi099 (mon.0) 493 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4cec51eb-d537-428a-96ea-bfcf39581618"}]': finished 2024-01-15T22:31:17.162 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:16 smithi162 conmon[39755]: cluster 2024-01-15T22:31:16.606732+0000 mon.smithi099 (mon.0) 494 : cluster [DBG] osdmap e22: 1 total, 0 up, 1 in 2024-01-15T22:31:17.162 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:16 smithi162 conmon[39755]: audit 2024-01-15T22:31:16.606871+0000 mon.smithi099 (mon.0) 495 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-15T22:31:17.921 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:31:17.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: cluster 2024-01-15T22:31:16.110520+0000 mgr.smithi099.cmjeyn (mgr.14184) 152 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:17.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: 2024-01-15T22:31:17.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: audit 2024-01-15T22:31:16.895131+0000 mon.smithi162 (mon.1) 13 : audit [INF] from='client.? 172.21.15.162:0/547372859' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:31:17.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: audit 2024-01-15T22:31:16.895743+0000 mon.smithi099 (mon.0) 496 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:31:17.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: audit 2024-01-15T22:31:16.901648+0000 mon.smithi099 (mon.0) 497 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-15T22:31:17.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: cluster 2024-01-15T22:31:16.901693+0000 mon.smithi099 (mon.0) 498 : cluster [DBG] osdmap e23: 0 total, 0 up, 0 in 2024-01-15T22:31:17.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: audit 2024-01-15T22:31:17.215115+0000 mon.smithi099 (mon.0) 499 : audit [INF] from='client.? 172.21.15.99:0/2470867757' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76a3f105-afde-43f8-887d-72b7c0f8df53"}]: dispatch 2024-01-15T22:31:17.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: audit 2024-01-15T22:31:17.222976+0000 mon.smithi099 (mon.0) 500 : audit [INF] from='client.? 172.21.15.99:0/2470867757' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "76a3f105-afde-43f8-887d-72b7c0f8df53"}]': finished 2024-01-15T22:31:17.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: cluster 2024-01-15T22:31:17.223015+0000 mon.smithi099 (mon.0) 501 : cluster [DBG] osdmap e24: 1 total, 0 up, 1 in 2024-01-15T22:31:17.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: audit 2024-01-15T22:31:17.223107+0000 mon.smithi099 (mon.0) 502 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-15T22:31:17.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: audit 2024-01-15T22:31:17.525331+0000 mon.smithi099 (mon.0) 503 : audit [INF] from='client.? 172.21.15.99:0/3518761235' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:31:17.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: audit 2024-01-15T22:31:17.533481+0000 mon.smithi099 (mon.0) 504 : audit [INF] from='client.? 172.21.15.99:0/3518761235' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-15T22:31:17.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: cluster 2024-01-15T22:31:17.533533+0000 mon.smithi099 (mon.0) 505 : cluster [DBG] osdmap e25: 0 total, 0 up, 0 in 2024-01-15T22:31:17.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: cephadm 2024-01-15T22:31:17.643461+0000 mgr.smithi099.cmjeyn (mgr.14184) 153 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi162', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-15T22:31:17.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:31:17.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:31:17.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: return f(*arg) 2024-01-15T22:31:17.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:31:17.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:31:17.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:31:17.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: code, '\n'.join(err))) 2024-01-15T22:31:17.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:31:17.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcaw1ycal:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpaizqcjrk:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:31:17.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:31:17.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:31:17.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:31:17.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4cec51eb-d537-428a-96ea-bfcf39581618 2024-01-15T22:31:17.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:31:17.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:31:17.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:31:17.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:31:17.950 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:31:17.951 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr self.prepare() 2024-01-15T22:31:17.951 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:31:17.951 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:31:17.951 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:31:17.951 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:31:17.951 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:31:17.951 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:31:17.951 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:31:17.951 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:31:17.951 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:31:17.951 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:31:17.951 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:31:17.951 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:31:17.951 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:31:17.951 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:31:17.951 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:31:17.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:31:17.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:31:17.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:31:17.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:31:17.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:31:17.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:31:17.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:31:17.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:31:17.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:31:17.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:31:17.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:31:17.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:31:17.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:31:17.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:31:17.952 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:31:17.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:31:17.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:31:17.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:31:17.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:31:17.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:31:17.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:31:17.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr prepare_step.safe_ 2024-01-15T22:31:17.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: prepare(args) 2024-01-15T22:31:17.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:31:17.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:31:17.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:31:17.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:31:17.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:31:17.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:31:17.953 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:31:17.954 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:31:17.954 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:31:17.954 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:31:17.954 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:31:17.954 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:31:17.954 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:31:17.954 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:31:17.954 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:31:17.954 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: main() 2024-01-15T22:31:17.954 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:31:17.954 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: r = ctx.func(ctx) 2024-01-15T22:31:17.954 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:31:17.954 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:31:17.954 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:31:17.955 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:31:17.955 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:31:17.955 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:31:17.955 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:31:17.955 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:31:17.955 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:31:17.955 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:31:17.955 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:31:17.955 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:31:17.955 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:17 smithi162 conmon[39755]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcaw1ycal:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpaizqcjrk:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:31:18.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: cluster 2024-01-15T22:31:16.110520+0000 mgr.smithi099.cmjeyn ( 2024-01-15T22:31:18.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: mgr.14184) 152 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:18.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: audit 2024-01-15T22:31:16.895131+0000 mon.smithi162 (mon.1) 13 : 2024-01-15T22:31:18.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: audit [INF] from='client.? 172.21.15.162:0/547372859' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:31:18.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: audit 2024-01 2024-01-15T22:31:18.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: -15T22:31:16.895743+0000 mon.smithi099 (mon.0) 496 : audit [INF] 2024-01-15T22:31:18.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:31:18.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: audit 2024-01-15T22:31:16.901648+0000 2024-01-15T22:31:18.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: mon.smithi099 (mon.0) 497 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-15T22:31:18.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: 2024-01-15T22:31:18.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: cluster 2024-01-15T22:31:18.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: 2024-01-15T22:31: 2024-01-15T22:31:18.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: 16.901693+0000 mon.smithi099 (mon.0) 498 : cluster 2024-01-15T22:31:18.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: [DBG] osdmap e23: 0 total, 0 up, 0 in 2024-01-15T22:31:18.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: audit 2024-01- 2024-01-15T22:31:18.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: 15T22:31:17.215115+0000 mon.smithi099 (mon. 2024-01-15T22:31:18.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: 0) 499 : audit [INF] from='client.? 172.21.15.99:0/2470867757' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76a3f105-afde-43f8-887d-72b7c0f8df53"}]: dispatch 2024-01-15T22:31:18.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: audit 2024-01-15T 2024-01-15T22:31:18.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: 22:31:17.222976+0000 mon.smithi099 (mon.0) 2024-01-15T22:31:18.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: 500 : audit [INF] from='client.? 172.21.15.99:0/2470867757' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "76a3f105-afde-43f8-887d-72b7c0f8df53"}]': finished 2024-01-15T22:31:18.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: cluster 2024-01-15T22 2024-01-15T22:31:18.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: :31:17.223015+0000 mon.smithi099 (mon.0) 501 2024-01-15T22:31:18.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: : cluster [DBG] osdmap e24: 1 total, 0 up, 1 in 2024-01-15T22:31:18.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: audit 2024-01-15T22:31 2024-01-15T22:31:18.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: :17.223107+0000 mon.smithi099 (mon.0) 502 : audit 2024-01-15T22:31:18.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-15T22:31:18.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: audit 2024-01-15T22:31:17. 2024-01-15T22:31:18.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: 525331+0000 mon.smithi099 (mon.0) 503 : audit [INF] 2024-01-15T22:31:18.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: from='client.? 172.21.15.99:0/3518761235' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:31:18.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: audit 2024-01-15T22:31:17.533481+0000 mon.smithi099 2024-01-15T22:31:18.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: (mon.0) 504 : audit [INF] from='client.? 172.21.15.99:0/3518761235' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-15T22:31:18.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: cluster 2024-01- 2024-01-15T22:31:18.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: 15T22:31:17.533533+0000 mon.smithi099 (mon. 2024-01-15T22:31:18.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: 0) 505 : cluster [DBG] osdmap e25: 0 total, 0 up, 0 in 2024-01-15T22:31:18.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: cephadm 2024- 2024-01-15T22:31:18.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: 01-15T22:31:17.643461+0000 mgr.smithi099.cmjeyn (mgr 2024-01-15T22:31:18.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: .14184) 153 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi162', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-15T22:31:18.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:31:18.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:31:18.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: return f(*arg) 2024-01-15T22:31:18.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:31:18.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:31:18.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:31:18.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: code, '\n'.join(err))) 2024-01-15T22:31:18.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:31:18.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcaw1ycal:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpaizqcjrk:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:31:18.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:31:18.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:31:18.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:31:18.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4cec51eb-d537-428a-96ea-bfcf39581618 2024-01-15T22:31:18.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:31:18.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:31:18.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:31:18.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:31:18.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:31:18.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr self.prepare() 2024-01-15T22:31:18.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:31:18.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:31:18.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:31:18.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:31:18.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:31:18.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:31:18.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:31:18.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:31:18.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:31:18.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:31:18.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:31:18.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:31:18.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:31:18.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:31:18.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:31:18.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:31:18.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:31:18.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:31:18.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:31:18.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:31:18.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:31:18.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:31:18.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:31:18.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:31:18.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:31:18.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:31:18.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:31:18.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:31:18.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:31:18.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:31:18.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:31:18.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:31:18.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:31:18.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:31:18.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:31:18.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:31:18.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:31:18.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:31:18.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:31:18.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:31:18.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:31:18.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:31:18.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:31:18.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:31:18.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:31:18.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:31:18.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:31:18.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:31:18.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:31:18.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:31:18.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:31:18.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:31:18.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: main() 2024-01-15T22:31:18.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:31:18.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: r = ctx.func(ctx) 2024-01-15T22:31:18.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:31:18.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:31:18.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:31:18.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:31:18.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:31:18.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:31:18.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf 2024-01-15T22:31:18.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: 565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:31:18.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:31:18.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:31:18.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:31:18.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:31:18.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:31:18.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:17 smithi099 conmon[35292]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcaw1ycal:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpaizqcjrk:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:31:18.776 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:31:19.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: audit 2024-01-15T22:31:17.921340+0000 mon.smithi099 (mon.0 2024-01-15T22:31:19.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: ) 506 : audit [DBG] from='client.? 172.21.15.99:0/188634897' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:31:19.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: cluster 2024-01-15T22:31:18.110742+0000 mgr.smithi099.cmjeyn (mgr.14184) 154 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:19.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: cephadm 2024-01-15T22:31:19.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: 2024-01-15T22:31:18.602065+0000 mgr.smithi099.cmjeyn (mgr.14184) 155 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi162', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-15T22:31:19.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:31:19.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:31:19.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: return f(*arg) 2024-01-15T22:31:19.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:31:19.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:31:19.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:31:19.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: code, '\n'.join(err))) 2024-01-15T22:31:19.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi099/config 2024-01-15T22:31:19.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpl7hza76d:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpte27bfjs:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:31:19.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:31:19.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:31:19.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:31:19.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 76a3f105-afde-43f8-887d-72b7c0f8df53 2024-01-15T22:31:19.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:31:19.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:31:19.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:31:19.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:31:19.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:31:19.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr self.prepare() 2024-01-15T22:31:19.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:31:19.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:31:19.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:31:19.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:31:19.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:31:19.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:31:19.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:31:19.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:31:19.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:31:19.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:31:19.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:31:19.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:31:19.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:31:19.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:31:19.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:31:19.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:31:19.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:31:19.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:31:19.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:31:19.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:31:19.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:31:19.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:31:19.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:31:19.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:31:19.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:31:19.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:31:19.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:31:19.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:31:19.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:31:19.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:31:19.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:31:19.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:31:19.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:31:19.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:31:19.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:31:19.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:31:19.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:31:19.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:31:19.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:31:19.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:31:19.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:31:19.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:31:19.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:31:19.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:31:19.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:31:19.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:31:19.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:31:19.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:31:19.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:31:19.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:31:19.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:31:19.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:31:19.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: main() 2024-01-15T22:31:19.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:31:19.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: r = ctx.func(ctx) 2024-01-15T22:31:19.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:31:19.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:31:19.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:31:19.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:31:19.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:31:19.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:31:19.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a71 2024-01-15T22:31:19.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: 5/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:31:19.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:31:19.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:31:19.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:31:19.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:31:19.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:31:19.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpl7hza76d:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpte27bfjs:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:31:19.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: cephadm 2024-01-15T22:31:18.603859+0000 mgr.smithi099.cmjeyn (mgr.14184) 156 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-15T22:31:19.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: service_id: all-available-devices 2024-01-15T22:31:19.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: service_name: osd.all-available-devices 2024-01-15T22:31:19.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: placement: 2024-01-15T22:31:19.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: host_pattern: '*' 2024-01-15T22:31:19.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: spec: 2024-01-15T22:31:19.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: data_devices: 2024-01-15T22:31:19.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: all: true 2024-01-15T22:31:19.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: filter_logic: AND 2024-01-15T22:31:19.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: objectstore: bluestore 2024-01-15T22:31:19.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:31:19.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcaw1ycal:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpaizqcjrk:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:31:19.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:31:19.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:31:19.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:31:19.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4cec51eb-d537-428a-96ea-bfcf39581618 2024-01-15T22:31:19.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:31:19.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:31:19.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:31:19.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:31:19.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:31:19.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr self.prepare() 2024-01-15T22:31:19.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:31:19.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:31:19.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:31:19.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:31:19.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:31:19.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:31:19.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:31:19.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:31:19.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:31:19.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:31:19.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:31:19.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:31:19.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:31:19.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:31:19.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:31:19.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:31:19.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:31:19.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:31:19.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:31:19.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:31:19.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:31:19.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:31:19.166 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:31:19.166 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:31:19.166 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:31:19.166 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:31:19.166 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:31:19.166 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:31:19.166 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:31:19.166 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:31:19.166 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:31:19.166 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:31:19.166 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:31:19.166 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:31:19.166 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:31:19.166 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:31:19.166 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:31:19.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:31:19.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:31:19.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:31:19.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:31:19.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:31:19.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:31:19.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:31:19.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:31:19.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap 2024-01-15T22:31:19.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: .py", line 302, in zap_osd 2024-01-15T22:31:19.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:31:19.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:31:19.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:31:19.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:31:19.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:31:19.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:31:19.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: main() 2024-01-15T22:31:19.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:31:19.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: r = ctx.func(ctx) 2024-01-15T22:31:19.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:31:19.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:31:19.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:31:19.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:31:19.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:31:19.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:31:19.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:31:19.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:31:19.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:31:19.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:31:19.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:31:19.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:31:19.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcaw1ycal:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpaizqcjrk:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:31:19.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:31:19.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-15T22:31:19.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: if self._apply_service(spec): 2024-01-15T22:31:19.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-15T22:31:19.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-15T22:31:19.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-15T22:31:19.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-15T22:31:19.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-15T22:31:19.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-15T22:31:19.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-15T22:31:19.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-15T22:31:19.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-15T22:31:19.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: raise self._value 2024-01-15T22:31:19.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-15T22:31:19.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: result = (True, func(*args, **kwds)) 2024-01-15T22:31:19.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-15T22:31:19.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: return list(map(*args)) 2024-01-15T22:31:19.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:31:19.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: return f(*arg) 2024-01-15T22:31:19.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:31:19.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:31:19.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:31:19.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: code, '\n'.join(err))) 2024-01-15T22:31:19.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:31:19.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcaw1ycal:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpaizqcjrk:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:31:19.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:31:19.171 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:31:19.171 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:31:19.171 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4cec51eb-d537-428a-96ea-bfcf39581618 2024-01-15T22:31:19.171 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:31:19.171 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:31:19.171 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:31:19.171 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:31:19.171 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:31:19.171 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr self.prepare() 2024-01-15T22:31:19.171 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:31:19.171 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:31:19.171 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:31:19.171 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:31:19.171 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:31:19.171 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:31:19.172 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:31:19.172 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:31:19.172 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:31:19.172 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:31:19.172 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:31:19.172 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:31:19.172 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:31:19.172 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:31:19.172 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:31:19.172 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:31:19.172 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:31:19.172 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File " 2024-01-15T22:31:19.172 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:31:19.172 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:31:19.172 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:31:19.173 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:31:19.173 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:31:19.173 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:31:19.173 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:31:19.173 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:31:19.173 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:31:19.173 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:31:19.173 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:31:19.173 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:31:19.173 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:31:19.173 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:31:19.173 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:31:19.173 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:31:19.173 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:31:19.173 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:31:19.173 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:31:19.174 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:31:19.174 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:31:19.174 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:31:19.174 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:31:19.174 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:31:19.174 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:31:19.174 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:31:19.174 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:31:19.174 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:31:19.174 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:31:19.174 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:31:19.174 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:31:19.174 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:31:19.174 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:31:19.174 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:31:19.174 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:31:19.174 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: main() 2024-01-15T22:31:19.175 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:31:19.175 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: r = ctx.func(ctx) 2024-01-15T22:31:19.175 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:31:19.175 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:31:19.175 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:31:19.175 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:31:19.175 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:31:19.175 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:31:19.175 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:31:19.175 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:31:19.175 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:31:19.175 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:31:19.175 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:31:19.175 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:31:19.175 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcaw1ycal:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpaizqcjrk:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:31:19.175 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:18 smithi099 conmon[35292]: cluster 2024-01-15T22:31:18.604530+0000 mgr.smithi099.cmjeyn (mgr.14184) 157 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:19.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: audit 2024-01-15T22:31:17.921340+0000 mon.smithi099 (mon.0) 506 : audit [DBG] from='client.? 172.21.15.99:0/188634897' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:31:19.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: cluster 2024-01-15T22:31:18.110742+0000 mgr.smithi099.cmjeyn (mgr.14184) 154 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:19.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: cephadm 2024-01-15T22:31:18.602065+0000 mgr.smithi099.cmjeyn (mgr. 2024-01-15T22:31:19.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: 14184) 155 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi162', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-15T22:31:19.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:31:19.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:31:19.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: return f(*arg) 2024-01-15T22:31:19.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:31:19.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:31:19.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:31:19.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: code, '\n'.join(err))) 2024-01-15T22:31:19.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi099/config 2024-01-15T22:31:19.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpl7hza76d:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpte27bfjs:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:31:19.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:31:19.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:31:19.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:31:19.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 76a3f105-afde-43f8-887d-72b7c0f8df53 2024-01-15T22:31:19.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:31:19.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:31:19.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:31:19.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:31:19.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:31:19.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr self.prepare() 2024-01-15T22:31:19.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:31:19.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:31:19.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:31:19.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:31:19.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:31:19.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:31:19.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:31:19.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:31:19.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:31:19.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:31:19.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:31:19.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:31:19.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:31:19.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:31:19.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:31:19.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:31:19.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:31:19.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:31:19.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:31:19.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:31:19.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:31:19.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:31:19.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:31:19.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:31:19.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:31:19.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:31:19.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:31:19.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:31:19.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:31:19.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:31:19.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:31:19.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:31:19.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:31:19.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:31:19.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:31:19.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:31:19.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:31:19.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:31:19.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:31:19.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:31:19.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:31:19.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:31:19.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:31:19.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:31:19.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:31:19.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:31:19.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:31:19.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:31:19.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:31:19.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:31:19.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:31:19.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:31:19.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: main() 2024-01-15T22:31:19.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:31:19.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: r = ctx.func(ctx) 2024-01-15T22:31:19.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:31:19.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:31:19.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:31:19.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:31:19.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:31:19.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:31:19.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf5 2024-01-15T22:31:19.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: 65308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:31:19.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:31:19.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:31:19.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:31:19.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:31:19.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:31:19.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpl7hza76d:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpte27bfjs:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:31:19.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: cephadm 2024-01-15T22:31:18.603859+0000 mgr.smithi099.cmjeyn (mgr.14184) 156 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-15T22:31:19.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: service_id: all-available-devices 2024-01-15T22:31:19.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: service_name: osd.all-available-devices 2024-01-15T22:31:19.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: placement: 2024-01-15T22:31:19.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: host_pattern: '*' 2024-01-15T22:31:19.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: spec: 2024-01-15T22:31:19.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: data_devices: 2024-01-15T22:31:19.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: all: true 2024-01-15T22:31:19.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: filter_logic: AND 2024-01-15T22:31:19.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: objectstore: bluestore 2024-01-15T22:31:19.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:31:19.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcaw1ycal:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpaizqcjrk:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:31:19.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:31:19.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:31:19.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:31:19.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4cec51eb-d537-428a-96ea-bfcf39581618 2024-01-15T22:31:19.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:31:19.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:31:19.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:31:19.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:31:19.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:31:19.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr self.prepare() 2024-01-15T22:31:19.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:31:19.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:31:19.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:31:19.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:31:19.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:31:19.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:31:19.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:31:19.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:31:19.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:31:19.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:31:19.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:31:19.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:31:19.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:31:19.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:31:19.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:31:19.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:31:19.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:31:19.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:31:19.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:31:19.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:31:19.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:31:19.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:31:19.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:31:19.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:31:19.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:31:19.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:31:19.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:31:19.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:31:19.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:31:19.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:31:19.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:31:19.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:31:19.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:31:19.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:31:19.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:31:19.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:31:19.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:31:19.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:31:19.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:31:19.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:31:19.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:31:19.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:31:19.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:31:19.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:31:19.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:31:19.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:31:19.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr devices 2024-01-15T22:31:19.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:31:19.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:31:19.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:31:19.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:31:19.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:31:19.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:31:19.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: main() 2024-01-15T22:31:19.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:31:19.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: r = ctx.func(ctx) 2024-01-15T22:31:19.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:31:19.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:31:19.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:31:19.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:31:19.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:31:19.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:31:19.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:31:19.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:31:19.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:31:19.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:31:19.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:31:19.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:31:19.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcaw1ycal:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpaizqcjrk:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:31:19.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:31:19.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-15T22:31:19.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: if self._apply_service(spec): 2024-01-15T22:31:19.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-15T22:31:19.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-15T22:31:19.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-15T22:31:19.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-15T22:31:19.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-15T22:31:19.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-15T22:31:19.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-15T22:31:19.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-15T22:31:19.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-15T22:31:19.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: raise self._value 2024-01-15T22:31:19.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-15T22:31:19.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: result = (True, func(*args, **kwds)) 2024-01-15T22:31:19.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-15T22:31:19.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: return list(map(*args)) 2024-01-15T22:31:19.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:31:19.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: return f(*arg) 2024-01-15T22:31:19.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:31:19.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:31:19.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:31:19.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: code, '\n'.join(err))) 2024-01-15T22:31:19.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:31:19.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcaw1ycal:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpaizqcjrk:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:31:19.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:31:19.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:31:19.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:31:19.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 4cec51eb-d537-428a-96ea-bfcf39581618 2024-01-15T22:31:19.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:31:19.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:31:19.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:31:19.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:31:19.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:31:19.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr self.prepare() 2024-01-15T22:31:19.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:31:19.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:31:19.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:31:19.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:31:19.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:31:19.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:31:19.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:31:19.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:31:19.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:31:19.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:31:19.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:31:19.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:31:19.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:31:19.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:31:19.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:31:19.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:31:19.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:31:19.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py" 2024-01-15T22:31:19.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: , line 59, in newfunc 2024-01-15T22:31:19.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:31:19.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:31:19.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:31:19.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:31:19.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:31:19.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:31:19.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:31:19.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:31:19.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:31:19.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:31:19.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:31:19.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:31:19.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:31:19.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:31:19.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:31:19.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:31:19.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:31:19.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:31:19.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:31:19.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:31:19.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:31:19.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:31:19.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:31:19.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:31:19.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:31:19.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:31:19.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:31:19.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:31:19.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:31:19.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:31:19.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:31:19.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:31:19.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:31:19.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:31:19.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: main() 2024-01-15T22:31:19.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:31:19.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: r = ctx.func(ctx) 2024-01-15T22:31:19.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:31:19.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:31:19.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:31:19.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:31:19.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:31:19.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:31:19.219 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:31:19.219 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:31:19.219 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:31:19.219 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:31:19.219 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:31:19.219 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:31:19.219 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpcaw1ycal:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpaizqcjrk:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:31:19.219 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:18 smithi162 conmon[39755]: cluster 2024-01-15T22:31:18.604530+0000 mgr.smithi099.cmjeyn (mgr.14184) 157 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:19.777 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:31:19.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:19 smithi162 conmon[39755]: cluster 2024-01-15T22:31:18.817985+0000 mon.smithi099 (mon.0) 507 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-15T22:31:20.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:19 smithi099 conmon[35292]: cluster 2024-01-15T22:31:18.817985+0000 2024-01-15T22:31:20.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:19 smithi099 conmon[35292]: mon.smithi099 (mon.0) 507 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-15T22:31:21.213 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:20 smithi099 conmon[35292]: cluster 2024-01-15T22:31:20.604926+0000 mgr.smithi099.cmjeyn (mgr.14184) 158 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:21.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:20 smithi162 conmon[39755]: cluster 2024-01-15T22:31:20.604926+0000 mgr.smithi099.cmjeyn (mgr.14184) 158 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:21.532 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:31:22.092 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:31:22.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:21 smithi099 conmon[35292]: audit 2024-01-15T22:31:21.532167+0000 mon.smithi099 (mon.0) 508 2024-01-15T22:31:22.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:21 smithi099 conmon[35292]: : audit [DBG] from='client.? 172.21.15.99:0/2494231257' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:31:22.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:21 smithi162 conmon[39755]: audit 2024-01-15T22:31:21.532167+0000 mon.smithi099 (mon.0) 508 : audit [DBG] from='client.? 172.21.15.99:0/2494231257' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:31:23.093 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:31:23.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:22 smithi099 conmon[35292]: cluster 2024-01-15T22:31:22.605645+0000 mgr.smithi099.cmjeyn (mgr.14184) 159 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:23.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:22 smithi099 conmon[35292]: 2024-01-15T22:31:23.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:22 smithi162 conmon[39755]: cluster 2024-01-15T22:31:22.605645+0000 mgr.smithi099.cmjeyn (mgr.14184) 159 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:24.802 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:31:25.400 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:31:25.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:25 smithi162 conmon[39755]: cluster 2024-01-15T22:31:24.606057+0000 mgr.smithi099.cmjeyn (mgr.14184) 160 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:25.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:25 smithi162 conmon[39755]: audit 2024-01-15T22:31:25.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:25 smithi162 conmon[39755]: 2024-01-15T22:31:24.802143+0000 mon.smithi099 (mon.0) 509 : audit [DBG] from='client.? 172.21.15.99:0/2217087168' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:31:26.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:25 smithi099 conmon[35292]: cluster 2024-01-15T22:31:24.606057+0000 mgr.smithi099.cmjeyn (mgr.14184) 160 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:26.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:25 smithi099 conmon[35292]: audit 2024-01-15T22:31:24.802143+0000 mon.smithi099 (mon.0) 509 : audit [DBG] from='client.? 172.21.15.99:0/2217087168' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:31:26.401 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:31:27.857 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:27 smithi099 conmon[35292]: cluster 2024-01-15T22:31:26.606473+0000 mgr.smithi099.cmjeyn (mgr.14184) 2024-01-15T22:31:27.857 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:27 smithi099 conmon[35292]: 161 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:27.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:27 smithi162 conmon[39755]: cluster 2024-01-15T22:31:26.606473+0000 mgr.smithi099.cmjeyn (mgr.14184) 161 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:28.173 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:31:28.773 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:28 smithi099 conmon[35292]: audit 2024-01-15T22:31:28.173651+0000 mon.smithi099 (mon.0) 2024-01-15T22:31:28.773 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:28 smithi099 conmon[35292]: 510 : audit [DBG] from='client.? 172.21.15.99:0/3004477180' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:31:28.791 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:31:28.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:28 smithi162 conmon[39755]: audit 2024-01-15T22:31:28.173651+0000 mon.smithi099 (mon.0) 510 : audit [DBG] from='client.? 172.21.15.99:0/3004477180' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:31:29.792 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:31:29.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:29 smithi162 conmon[39755]: cluster 2024-01-15T22:31:28.606863+0000 mgr.smithi099.cmjeyn (mgr.14184) 162 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:30.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:29 smithi099 conmon[35292]: cluster 2024-01-15T22:31:28.606863+0000 mgr.smithi099.cmjeyn (mgr.14184) 162 : cluster [DBG] 2024-01-15T22:31:30.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:29 smithi099 conmon[35292]: pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:31.546 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:31:31.566 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:31 smithi099 conmon[35292]: audit 2024-01-15T22:31:30.355640+0000 mon.smithi099 (mon.0) 511 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:31:31.566 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:31 smithi099 conmon[35292]: cluster 2024-01-15T22:31:30.607220+0000 mgr.smithi099.cmjeyn (mgr.14184) 163 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:31.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:31 smithi162 conmon[39755]: audit 2024-01-15T22:31:30.355640+0000 mon.smithi099 (mon.0) 511 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:31:31.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:31 smithi162 conmon[39755]: cluster 2024-01-15T22:31:30.607220+0000 mgr.smithi099.cmjeyn (mgr.14184) 163 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:32.123 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:31:32.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:32 smithi099 conmon[35292]: audit 2024-01-15T22:31:31.545732+0000 mon.smithi099 (mon.0) 512 2024-01-15T22:31:32.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:32 smithi099 conmon[35292]: : audit [DBG] from='client.? 172.21.15.99:0/2330320985' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:31:32.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:32 smithi162 conmon[39755]: audit 2024-01-15T22:31:31.545732+0000 mon.smithi099 (mon.0) 512 : audit [DBG] from='client.? 172.21.15.99:0/2330320985' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:31:33.125 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:31:33.574 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:33 smithi099 conmon[35292]: cluster 2024-01-15T22:31:32.607581+0000 mgr.smithi099.cmjeyn (mgr.14184) 2024-01-15T22:31:33.574 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:33 smithi099 conmon[35292]: 164 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:33.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:33 smithi162 conmon[39755]: cluster 2024-01-15T22:31:32.607581+0000 mgr.smithi099.cmjeyn (mgr.14184) 164 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:34.972 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:31:35.538 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:31:35.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:35 smithi099 conmon[35292]: cluster 2024-01-15T22:31:34.607995+0000 mgr.smithi099.cmjeyn (mgr.14184) 165 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:35.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:35 smithi099 conmon[35292]: audit 2024-01-15T22:31:34.971770+0000 mon.smithi099 (mon.0) 513 : audit [DBG] from='client.? 172.21.15.99:0/3552919064' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:31:35.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:35 smithi162 conmon[39755]: cluster 2024-01-15T22:31:34.607995+0000 mgr.smithi099.cmjeyn (mgr.14184) 165 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:35.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:35 smithi162 conmon[39755]: audit 2024-01-15T22:31:34.971770+0000 mon.smithi099 (mon.0) 513 : audit [DBG] from='client.? 172.21.15.99:0/3552919064' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:31:36.540 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:31:37.933 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:37 smithi099 conmon[35292]: cluster 2024-01-15T22:31:36.608407+0000 mgr.smithi099.cmjeyn (mgr.14184) 166 : 2024-01-15T22:31:37.933 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:37 smithi099 conmon[35292]: cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:37.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:37 smithi162 conmon[39755]: cluster 2024-01-15T22:31:36.608407+0000 mgr.smithi099.cmjeyn (mgr.14184) 166 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:38.283 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:31:38.805 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:31:38.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:38 smithi162 conmon[39755]: audit 2024-01-15T22:31:38.282654+0000 mon.smithi099 (mon.0) 514 : audit [DBG] from='client.? 172.21.15.99:0/3660557069' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:31:39.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:38 smithi099 conmon[35292]: audit 2024-01-15T22:31:38.282654+0000 mon.smithi099 (mon.0) 514 : 2024-01-15T22:31:39.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:38 smithi099 conmon[35292]: audit [DBG] from='client.? 172.21.15.99:0/3660557069' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:31:39.806 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:31:39.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:39 smithi162 conmon[39755]: cluster 2024-01-15T22:31:38.608803+0000 mgr.smithi099.cmjeyn (mgr.14184) 167 : cluster 2024-01-15T22:31:39.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:39 smithi162 conmon[39755]: [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:40.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:39 smithi099 conmon[35292]: cluster 2024-01-15T22:31:38.608803+0000 mgr.smithi099.cmjeyn (mgr.14184) 167 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:41.214 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:40 smithi099 conmon[35292]: cluster 2024-01-15T22:31:40.609237+0000 mgr.smithi099.cmjeyn (mgr.14184) 168 2024-01-15T22:31:41.215 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:40 smithi099 conmon[35292]: : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:41.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:40 smithi162 conmon[39755]: cluster 2024-01-15T22:31:40.609237+0000 mgr.smithi099.cmjeyn (mgr.14184) 168 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:41.535 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:31:42.063 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:31:42.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:41 smithi099 conmon[35292]: audit 2024-01-15T22:31:41.534676+0000 mon.smithi099 (mon.0) 515 : audit [DBG] from='client.? 172.21.15.99:0/1705711400' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:31:42.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:41 smithi162 conmon[39755]: audit 2024-01-15T22:31:41.534676+0000 mon.smithi099 (mon.0) 515 : audit [DBG] from='client.? 172.21.15.99:0/1705711400' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:31:43.065 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:31:43.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:42 smithi099 conmon[35292]: cluster 2024-01-15T22:31:42.609702+0000 mgr.smithi099.cmjeyn (mgr.14184) 169 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:43.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:42 smithi162 conmon[39755]: cluster 2024-01-15T22:31:42.609702+0000 mgr.smithi099.cmjeyn (mgr.14184) 169 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:44.862 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:31:45.514 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:31:45.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:45 smithi099 conmon[35292]: cluster 2024-01-15T22:31:44.610125+0000 mgr.smithi099.cmjeyn (mgr.14184) 170 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:45.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:45 smithi099 conmon[35292]: audit 2024-01-15T22:31:44.861242+0000 mon.smithi099 (mon.0) 516 : audit [DBG] from='client.? 172.21.15.99:0/3679966129' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:31:45.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:45 smithi162 conmon[39755]: cluster 2024-01-15T22:31:44.610125+0000 mgr.smithi099.cmjeyn (mgr.14184) 170 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:45.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:45 smithi162 conmon[39755]: audit 2024-01-15T22:31:44.861242+0000 mon.smithi099 (mon.0) 516 : audit [DBG] from='client.? 172.21.15.99:0/3679966129' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:31:46.516 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:31:47.939 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:47 smithi099 conmon[35292]: cluster 2024-01-15T22:31:46.610858+0000 mgr.smithi099.cmjeyn (mgr.14184) 171 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:47.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:47 smithi162 conmon[39755]: cluster 2024-01-15T22:31:46.610858+0000 mgr.smithi099.cmjeyn (mgr.14184) 171 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:48.285 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:31:48.837 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:31:48.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:48 smithi162 conmon[39755]: audit 2024-01-15T22:31:48.285363+0000 mon.smithi099 (mon.0) 517 : audit [DBG] from='client.? 172.21.15.99:0/3725161026' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:31:49.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:48 smithi099 conmon[35292]: audit 2024-01-15T22:31:48.285363+0000 mon.smithi099 (mon.0) 517 : audit [DBG] from='client.? 172.21.15.99:0/3725161026' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:31:49.838 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:31:49.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:49 smithi162 conmon[39755]: cluster 2024-01-15T22:31:48.611519+0000 mgr.smithi099.cmjeyn (mgr.14184) 172 : cluster [DBG] 2024-01-15T22:31:49.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:49 smithi162 conmon[39755]: pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:50.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:49 smithi099 conmon[35292]: cluster 2024-01-15T22:31:48.611519+0000 mgr.smithi099.cmjeyn (mgr.14184) 172 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:51.246 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:50 smithi099 conmon[35292]: cluster 2024-01-15T22:31:50.611963+0000 mgr.smithi099.cmjeyn (mgr. 2024-01-15T22:31:51.246 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:50 smithi099 conmon[35292]: 14184) 173 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:51.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:50 smithi162 conmon[39755]: cluster 2024-01-15T22:31:50.611963+0000 mgr.smithi099.cmjeyn (mgr.14184) 173 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:51.588 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:31:52.119 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:31:52.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:51 smithi099 conmon[35292]: audit 2024-01-15T22:31:51.588580+0000 mon.smithi099 (mon.0) 518 : audit [DBG] from='client.? 172.21.15.99:0/2728829459' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:31:52.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:51 smithi162 conmon[39755]: audit 2024-01-15T22:31:51.588580+0000 mon.smithi099 (mon.0) 518 : audit [DBG] from='client.? 172.21.15.99:0/2728829459' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:31:53.120 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:31:53.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:52 smithi099 conmon[35292]: cluster 2024-01-15T22:31:52.612451+0000 mgr.smithi099.cmjeyn (mgr.14184) 174 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:53.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:52 smithi162 conmon[39755]: cluster 2024-01-15T22:31:52.612451+0000 mgr.smithi099.cmjeyn (mgr.14184) 174 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:54.901 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:31:55.501 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:31:55.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:55 smithi099 conmon[35292]: cluster 2024-01-15T22:31:54.612903+0000 mgr.smithi099.cmjeyn (mgr.14184) 175 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:55.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:55 smithi099 conmon[35292]: audit 2024-01-15T22:31:54.900939+0000 mon.smithi099 (mon.0) 519 : audit [DBG] from='client.? 172.21.15.99:0/1874396392' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:31:55.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:55 smithi162 conmon[39755]: cluster 2024-01-15T22:31:54.612903+0000 mgr.smithi099.cmjeyn (mgr.14184) 175 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:55.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:55 smithi162 conmon[39755]: audit 2024-01-15T22:31:54.900939+0000 mon.smithi099 (mon.0) 519 : audit [DBG] from='client.? 172.21.15.99:0/1874396392' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:31:56.502 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:31:57.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:57 smithi162 conmon[39755]: cluster 2024-01-15T22:31:56.613430+0000 mgr.smithi099.cmjeyn (mgr.14184) 176 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:57.962 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:57 smithi099 conmon[35292]: cluster 2024-01-15T22:31:56.613430+0000 mgr.smithi099.cmjeyn (mgr.14184 2024-01-15T22:31:57.962 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:57 smithi099 conmon[35292]: ) 176 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:31:58.301 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:31:58.858 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:31:58.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:58 smithi162 conmon[39755]: audit 2024-01-15T22:31:58.301000+0000 mon.smithi099 2024-01-15T22:31:58.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:58 smithi162 conmon[39755]: (mon.0) 520 : audit [DBG] from='client.? 172.21.15.99:0/266470035' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:31:59.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:58 smithi099 conmon[35292]: audit 2024-01-15T22:31:58.301000+0000 mon.smithi099 (mon.0) 520 : audit [DBG] from='client.? 172.21.15.99:0/266470035' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:31:59.859 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:31:59.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:59 smithi162 conmon[39755]: cluster 2024-01-15T22:31:58.613864+0000 mgr.smithi099.cmjeyn (mgr.14184) 177 : cluster [DBG] 2024-01-15T22:31:59.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:31:59 smithi162 conmon[39755]: pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:00.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:31:59 smithi099 conmon[35292]: cluster 2024-01-15T22:31:58.613864+0000 mgr.smithi099.cmjeyn (mgr.14184) 177 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:01.254 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:00 smithi099 conmon[35292]: cluster 2024-01-15T22:32:00.614215+0000 mgr.smithi099.cmjeyn (mgr.14184) 178 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:01.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:00 smithi162 conmon[39755]: cluster 2024-01-15T22:32:00.614215+0000 mgr.smithi099.cmjeyn (mgr.14184) 178 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:01.593 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:32:02.150 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:32:02.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:01 smithi099 conmon[35292]: audit 2024-01-15T22:32:01.593484+0000 mon.smithi099 (mon.0) 521 : audit [DBG] from='client.? 172.21.15.99:0/1790898246' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:32:02.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:01 smithi162 conmon[39755]: audit 2024-01-15T22:32:01.593484+0000 mon.smithi099 (mon.0) 521 : audit [DBG] from='client.? 172.21.15.99:0/1790898246' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:32:03.151 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:32:03.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:02 smithi099 conmon[35292]: cluster 2024-01-15T22:32:02.614567+0000 mgr.smithi099.cmjeyn (mgr.14184) 179 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:03.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:02 smithi162 conmon[39755]: cluster 2024-01-15T22:32:02.614567+0000 mgr.smithi099.cmjeyn (mgr.14184) 179 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:04.852 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:32:05.374 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:32:05.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:05 smithi162 conmon[39755]: cluster 2024-01-15T22:32:04.614975+0000 mgr.smithi099.cmjeyn (mgr.14184) 180 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:05.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:05 smithi162 conmon[39755]: 2024-01-15T22:32:05.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:05 smithi162 conmon[39755]: audit 2024-01-15T22:32:04.851115+0000 mon.smithi099 (mon.0) 522 : audit [DBG] from='client.? 172.21.15.99:0/1598522365' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:32:06.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:05 smithi099 conmon[35292]: cluster 2024-01-15T22:32:04.614975+0000 mgr.smithi099.cmjeyn (mgr.14184) 180 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:06.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:05 smithi099 conmon[35292]: audit 2024-01-15T22:32:04.851115+0000 mon.smithi099 (mon.0) 522 : audit [DBG] from='client.? 172.21.15.99:0/1598522365' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:32:06.375 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:32:07.851 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:07 smithi099 conmon[35292]: cluster 2024-01-15T22:32:06.615540+0000 mgr.smithi099.cmjeyn (mgr.14184) 181 2024-01-15T22:32:07.851 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:07 smithi099 conmon[35292]: : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:07.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:07 smithi162 conmon[39755]: cluster 2024-01-15T22:32:06.615540+0000 mgr.smithi099.cmjeyn (mgr.14184) 181 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:08.161 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:32:08.700 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:32:08.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:08 smithi099 conmon[35292]: audit 2024-01-15T22:32:08.160607+0000 mon.smithi099 (mon.0) 523 : audit [DBG] from='client.? 172.21.15.99:0/4039504838' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:32:08.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:08 smithi162 conmon[39755]: audit 2024-01-15T22:32:08.160607+0000 mon.smithi099 (mon.0) 523 : audit [DBG] from='client.? 172.21.15.99:0/4039504838' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:32:09.701 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:32:09.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:09 smithi162 conmon[39755]: cluster 2024-01-15T22:32:08.616169+0000 mgr.smithi099.cmjeyn (mgr.14184) 182 2024-01-15T22:32:09.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:09 smithi162 conmon[39755]: : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:10.135 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:09 smithi099 conmon[35292]: cluster 2024-01-15T22:32:08.616169+0000 mgr.smithi099.cmjeyn (mgr.14184) 182 : cluster [DBG] 2024-01-15T22:32:10.135 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:09 smithi099 conmon[35292]: pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:11.142 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:10 smithi099 conmon[35292]: cluster 2024-01-15T22:32:10.616622+0000 mgr.smithi099.cmjeyn (mgr.14184) 183 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:11.142 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:10 smithi099 conmon[35292]: 2024-01-15T22:32:11.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:10 smithi162 conmon[39755]: cluster 2024-01-15T22:32:10.616622+0000 mgr.smithi099.cmjeyn (mgr.14184) 183 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:11.455 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:32:12.006 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:32:12.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:11 smithi099 conmon[35292]: audit 2024-01-15T22:32:11.454871+0000 mon.smithi099 (mon.0) 524 : audit [DBG] from='client.? 172.21.15.99:0/2928307682' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:32:12.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:11 smithi162 conmon[39755]: audit 2024-01-15T22:32:11.454871+0000 mon.smithi099 (mon.0) 524 : audit [DBG] from='client.? 172.21.15.99:0/2928307682' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:32:13.007 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:32:13.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:12 smithi099 conmon[35292]: cluster 2024-01-15T22:32:12.617089+0000 mgr.smithi099.cmjeyn (mgr.14184) 184 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:13.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:12 smithi162 conmon[39755]: cluster 2024-01-15T22:32:12.617089+0000 mgr.smithi099.cmjeyn (mgr.14184) 184 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:14.751 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:32:15.329 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:32:15.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:15 smithi162 conmon[39755]: cluster 2024-01-15T22:32:14.617474+0000 mgr.smithi099.cmjeyn (mgr.14184) 185 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:15.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:15 smithi162 conmon[39755]: audit 2024-01-15T22:32:14.750676+0000 mon.smithi099 (mon.0) 525 : audit [DBG] from='client.? 172.21.15.99:0/2830461878' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:32:16.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:15 smithi099 conmon[35292]: cluster 2024-01-15T22:32:14.617474+0000 mgr.smithi099.cmjeyn (mgr.14184) 185 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:16.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:15 smithi099 conmon[35292]: audit 2024-01-15T22:32:14.750676+0000 mon.smithi099 (mon.0) 525 : audit [DBG] from='client.? 172.21.15.99:0/2830461878' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:32:16.330 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:32:17.810 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:17 smithi099 conmon[35292]: cluster 2024-01-15T22:32:16.617892+0000 mgr.smithi099.cmjeyn (mgr.14184) 2024-01-15T22:32:17.810 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:17 smithi099 conmon[35292]: 186 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:17.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:17 smithi162 conmon[39755]: cluster 2024-01-15T22:32:16.617892+0000 mgr.smithi099.cmjeyn (mgr.14184) 186 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:18.131 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:32:18.707 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:18 smithi099 conmon[35292]: audit 2024-01-15T22:32:18.131086+0000 mon.smithi099 (mon.0 2024-01-15T22:32:18.707 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:18 smithi099 conmon[35292]: ) 526 : audit [DBG] from='client.? 172.21.15.99:0/4088406084' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:32:18.707 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:18 smithi099 conmon[35292]: audit 2024-01-15T22:32:18.606370+0000 mon.smithi099 (mon.0) 527 : audit 2024-01-15T22:32:18.707 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:18 smithi099 conmon[35292]: [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-15T22:32:18.708 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:32:18.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:18 smithi162 conmon[39755]: audit 2024-01-15T22:32:18.131086+0000 mon.smithi099 (mon.0) 526 : audit [DBG] from='client.? 172.21.15.99:0/4088406084' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:32:18.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:18 smithi162 conmon[39755]: audit 2024-01-15T22:32:18.606370+0000 mon.smithi099 (mon.0) 527 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-15T22:32:19.709 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:32:19.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:19 smithi099 conmon[35292]: cluster 2024-01-15T22:32:18.618403+0000 mgr.smithi099.cmjeyn (mgr.14184) 187 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:19.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:19 smithi099 conmon[35292]: audit 2024-01-15T22:32:18.933268+0000 mon.smithi099 (mon.0) 528 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:32:19.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:19 smithi099 conmon[35292]: audit 2024-01-15T22:32:18.933916+0000 mon.smithi099 (mon.0) 529 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-15T22:32:19.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:19 smithi099 conmon[35292]: audit 2024-01-15T22:32:18.938251+0000 mon.smithi099 (mon.0) 530 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:32:19.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:19 smithi099 conmon[35292]: audit 2024-01-15T22:32:18.941547+0000 mon.smithi099 (mon.0) 531 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-15T22:32:19.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:19 smithi099 conmon[35292]: audit 2024-01-15T22:32:18.944085+0000 mon.smithi099 (mon.0) 532 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-15T22:32:19.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:19 smithi099 conmon[35292]: audit 2024-01-15T22:32:18.945008+0000 mon.smithi099 (mon.0) 533 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-15T22:32:19.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:19 smithi099 conmon[35292]: audit 2024-01-15T22:32:18.945206+0000 mon.smithi099 (mon.0) 534 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:32:19.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:19 smithi099 conmon[35292]: audit 2024-01-15T22:32:18.945638+0000 mon.smithi099 (mon.0) 535 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:32:19.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:19 smithi162 conmon[39755]: cluster 2024-01-15T22:32:18.618403+0000 mgr.smithi099.cmjeyn (mgr.14184) 187 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:19.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:19 smithi162 conmon[39755]: audit 2024-01-15T22:32:18.933268+0000 mon.smithi099 (mon.0) 528 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:32:19.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:19 smithi162 conmon[39755]: audit 2024-01-15T22:32:18.933916+0000 mon.smithi099 (mon.0) 529 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-15T22:32:19.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:19 smithi162 conmon[39755]: audit 2024-01-15T22:32:18.938251+0000 mon.smithi099 (mon.0) 530 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:32:19.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:19 smithi162 conmon[39755]: audit 2024-01-15T22:32:18.941547+0000 mon.smithi099 (mon.0) 531 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-15T22:32:19.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:19 smithi162 conmon[39755]: audit 2024-01-15T22:32:18.944085+0000 mon.smithi099 (mon.0) 532 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-15T22:32:19.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:19 smithi162 conmon[39755]: audit 2024-01-15T22:32:18.945008+0000 mon.smithi099 (mon.0) 533 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-15T22:32:19.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:19 smithi162 conmon[39755]: audit 2024-01-15T22:32:18.945206+0000 mon.smithi099 (mon.0) 534 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:32:19.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:19 smithi162 conmon[39755]: audit 2024-01-15T22:32:18.945638+0000 mon.smithi099 (mon.0) 535 : audit 2024-01-15T22:32:19.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:19 smithi162 conmon[39755]: [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:32:20.831 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:20 smithi099 conmon[35292]: cluster 2024-01-15T22:32:18.938540+0000 mgr.smithi099.cmjeyn (mgr.14184) 188 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:20.831 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:20 smithi099 conmon[35292]: cluster 2024-01-15T22:32:19.936445+0000 mon.smithi099 (mon.0) 536 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-15T22:32:20.831 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:20 smithi099 conmon[35292]: cluster 2024-01-15T22:32:19.936477+0000 mon.smithi099 (mon.0) 537 : cluster [INF] Cluster is now healthy 2024-01-15T22:32:20.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:20 smithi162 conmon[39755]: cluster 2024-01-15T22:32:18.938540+0000 mgr.smithi099.cmjeyn (mgr.14184) 188 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:20.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:20 smithi162 conmon[39755]: cluster 2024-01-15T22:32:19.936445+0000 mon.smithi099 2024-01-15T22:32:20.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:20 smithi162 conmon[39755]: (mon.0) 536 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-15T22:32:20.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:20 smithi162 conmon[39755]: cluster 2024-01-15T22:32:19.936477+0000 mon.smithi099 (mon.0) 537 : cluster [INF] Cluster is now healthy 2024-01-15T22:32:21.907 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:32:22.089 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:21 smithi162 conmon[39755]: audit 2024-01-15T22:32:21.518539+0000 mon.smithi162 (mon.1) 2024-01-15T22:32:22.089 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:21 smithi162 conmon[39755]: 14 : audit [INF] from='client.? 172.21.15.162:0/2229530145' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8a59035-6972-4653-bd6d-383fb2963762"}]: dispatch 2024-01-15T22:32:22.090 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:21 smithi162 conmon[39755]: audit 2024-01-15T22:32:21.518948+0000 mon.smithi099 (mon.0) 538 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8a59035-6972-4653-bd6d-383fb2963762"}]: dispatch 2024-01-15T22:32:22.090 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:21 smithi162 conmon[39755]: audit 2024-01-15T22:32:21.524926+0000 mon.smithi099 (mon.0) 539 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e8a59035-6972-4653-bd6d-383fb2963762"}]': finished 2024-01-15T22:32:22.090 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:21 smithi162 conmon[39755]: cluster 2024-01-15T22:32:21.524983+0000 mon.smithi099 (mon.0) 540 : cluster [DBG] osdmap e26: 1 total, 0 up, 1 in 2024-01-15T22:32:22.090 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:21 smithi162 conmon[39755]: audit 2024-01-15T22:32:21.525081+0000 mon.smithi099 (mon.0) 541 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-15T22:32:22.097 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:21 smithi099 conmon[35292]: audit 2024-01-15T22:32:21.518539+0000 mon.smithi162 (mon.1) 14 : audit [INF] from='client.? 172.21.15.162:0/2229530145' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8a59035-6972-4653-bd6d-383fb2963762"}]: dispatch 2024-01-15T22:32:22.097 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:21 smithi099 conmon[35292]: audit 2024-01-15T22:32:21.518948+0000 mon.smithi099 (mon.0) 538 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8a59035-6972-4653-bd6d-383fb2963762"}]: dispatch 2024-01-15T22:32:22.097 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:21 smithi099 conmon[35292]: audit 2024-01-15T22:32:21.524926+0000 mon.smithi099 (mon.0) 539 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e8a59035-6972-4653-bd6d-383fb2963762"}]': finished 2024-01-15T22:32:22.097 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:21 smithi099 conmon[35292]: cluster 2024-01-15T22:32:21.524983+0000 mon.smithi099 (mon.0) 540 : cluster [DBG] osdmap e26: 1 total, 0 up, 1 in 2024-01-15T22:32:22.097 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:21 smithi099 conmon[35292]: audit 2024-01-15T22:32:21.525081+0000 mon.smithi099 (mon.0) 541 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-15T22:32:22.605 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":27,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:32:22.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:22 smithi099 conmon[35292]: cluster 2024-01-15T22:32:20.938892+0000 mgr.smithi099.cmjeyn (mgr.14184) 189 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:22.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:22 smithi099 conmon[35292]: audit 2024-01-15T22:32:21.814314+0000 mon.smithi162 (mon.1) 15 : audit [INF] from='client.? 172.21.15.162:0/1835676546' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:32:22.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:22 smithi099 conmon[35292]: audit 2024-01-15T22:32:21.814695+0000 mon.smithi099 (mon.0) 542 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:32:22.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:22 smithi099 conmon[35292]: audit 2024-01-15T22:32:21.823825+0000 mon.smithi099 (mon.0) 543 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-15T22:32:22.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:22 smithi099 conmon[35292]: cluster 2024-01-15T22:32:21.823908+0000 mon.smithi099 (mon.0) 544 : cluster [DBG] osdmap e27: 0 total, 0 up, 0 in 2024-01-15T22:32:22.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:22 smithi099 conmon[35292]: audit 2024-01-15T22:32:21.906659+0000 mon.smithi099 (mon.0) 545 : audit [DBG] from='client.? 172.21.15.99:0/3127915292' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:32:22.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:22 smithi099 conmon[35292]: audit 2024-01-15T22:32:22.130018+0000 mon.smithi099 (mon.0) 546 : audit [INF] from='client.? 172.21.15.99:0/3196566252' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea0d0325-0bb3-4cdd-ae49-9716f1cfb1cc"}]: dispatch 2024-01-15T22:32:22.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:22 smithi099 conmon[35292]: audit 2024-01-15T22:32:22.137987+0000 mon.smithi099 (mon.0) 547 : audit [INF] from='client.? 172.21.15.99:0/3196566252' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ea0d0325-0bb3-4cdd-ae49-9716f1cfb1cc"}]': finished 2024-01-15T22:32:22.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:22 smithi099 conmon[35292]: cluster 2024-01-15T22:32:22.138041+0000 mon.smithi099 (mon.0) 548 : cluster [DBG] osdmap e28: 1 total, 0 up, 1 in 2024-01-15T22:32:22.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:22 smithi099 conmon[35292]: audit 2024-01-15T22:32:22.138153+0000 mon.smithi099 (mon.0) 549 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-15T22:32:22.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:22 smithi099 conmon[35292]: audit 2024-01-15T22:32:22.427674+0000 mon.smithi099 (mon.0) 550 : audit [INF] from='client.? 172.21.15.99:0/3486800650' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:32:22.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:22 smithi099 conmon[35292]: audit 2024-01-15T22:32:22.435672+0000 mon.smithi099 (mon.0) 551 : audit [INF] from='client.? 172.21.15.99:0/3486800650' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-15T22:32:22.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:22 smithi099 conmon[35292]: cluster 2024-01-15T22:32:22.435714+0000 mon.smithi099 (mon.0) 552 : cluster [DBG] osdmap e29: 0 total, 0 up, 0 in 2024-01-15T22:32:22.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:22 smithi162 conmon[39755]: cluster 2024-01-15T22:32:20.938892+0000 mgr.smithi099.cmjeyn (mgr.14184) 189 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:22.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:22 smithi162 conmon[39755]: audit 2024-01-15T22:32:21.814314+0000 mon.smithi162 (mon.1) 15 : audit [INF] from='client.? 172.21.15.162:0/1835676546' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:32:22.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:22 smithi162 conmon[39755]: audit 2024-01-15T22:32:21.814695+0000 mon.smithi099 (mon.0) 542 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:32:22.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:22 smithi162 conmon[39755]: audit 2024-01-15T22:32:21.823825+0000 mon.smithi099 (mon.0) 543 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-15T22:32:22.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:22 smithi162 conmon[39755]: cluster 2024-01-15T22:32:21.823908+0000 mon.smithi099 (mon.0) 544 : cluster [DBG] osdmap e27: 0 total, 0 up, 0 in 2024-01-15T22:32:22.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:22 smithi162 conmon[39755]: audit 2024-01-15T22:32:21.906659+0000 mon.smithi099 (mon.0) 545 : audit [DBG] from='client.? 172.21.15.99:0/3127915292' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:32:22.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:22 smithi162 conmon[39755]: audit 2024-01-15T22:32:22.130018+0000 mon.smithi099 (mon.0) 546 : audit [INF] from='client.? 172.21.15.99:0/3196566252' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea0d0325-0bb3-4cdd-ae49-9716f1cfb1cc"}]: dispatch 2024-01-15T22:32:22.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:22 smithi162 conmon[39755]: audit 2024-01-15T22:32:22.137987+0000 mon.smithi099 (mon.0) 547 : audit [INF] from='client.? 172.21.15.99:0/3196566252' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ea0d0325-0bb3-4cdd-ae49-9716f1cfb1cc"}]': finished 2024-01-15T22:32:22.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:22 smithi162 conmon[39755]: cluster 2024-01-15T22 2024-01-15T22:32:22.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:22 smithi162 conmon[39755]: :32:22.138041+0000 mon.smithi099 (mon.0) 548 : cluster [DBG] osdmap e28: 1 total, 0 up, 1 in 2024-01-15T22:32:22.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:22 smithi162 conmon[39755]: audit 2024-01-15T22:32:22.138153+0000 mon.smithi099 (mon.0) 549 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-15T22:32:22.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:22 smithi162 conmon[39755]: audit 2024-01-15T22:32:22.427674+0000 mon.smithi099 (mon.0) 550 : audit [INF] from='client.? 172.21.15.99:0/3486800650' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:32:22.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:22 smithi162 conmon[39755]: audit 2024-01-15T22:32:22.435672+0000 mon.smithi099 (mon.0) 551 : audit [INF] from='client.? 172.21.15.99:0/3486800650' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-15T22:32:22.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:22 smithi162 conmon[39755]: cluster 2024-01-15T22:32:22.435714+0000 mon.smithi099 (mon.0) 552 : cluster [DBG] osdmap e29: 0 total, 0 up, 0 in 2024-01-15T22:32:23.606 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:32:23.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: cephadm 2024-01-15T22:32:22.528679+0000 mgr.smithi099.cmjeyn (mgr.14184) 190 2024-01-15T22:32:23.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi162', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-15T22:32:23.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:32:23.906 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:32:23.906 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: return f(*arg) 2024-01-15T22:32:23.906 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:32:23.906 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:32:23.906 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:32:23.906 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: code, '\n'.join(err))) 2024-01-15T22:32:23.906 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:32:23.906 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpwgq2u0iu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp2qfhqzgr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:32:23.906 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:32:23.906 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:32:23.906 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:32:23.906 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new e8a59035-6972-4653-bd6d-383fb2963762 2024-01-15T22:32:23.906 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:32:23.906 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:32:23.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:32:23.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:32:23.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:32:23.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr self.prepare() 2024-01-15T22:32:23.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:32:23.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:32:23.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:32:23.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:32:23.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:32:23.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:32:23.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:32:23.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:32:23.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:32:23.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:32:23.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:32:23.907 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:32:23.908 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:32:23.908 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:32:23.908 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:32:23.908 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:32:23.908 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:32:23.908 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:32:23.908 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:32:23.908 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:32:23.908 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:32:23.908 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:32:23.908 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:32:23.908 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:32:23.908 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:32:23.908 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:32:23.908 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:32:23.909 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:32:23.909 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:32:23.909 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:32:23.909 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:32:23.909 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:32:23.909 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:32:23.909 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:32:23.909 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:32:23.909 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:32:23.909 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:32:23.909 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:32:23.909 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:32:23.909 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:32:23.909 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:32:23.909 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:32:23.910 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:32:23.910 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:32:23.910 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:32:23.910 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:32:23.910 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:32:23.910 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:32:23.910 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:32:23.910 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:32:23.910 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:32:23.910 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:32:23.910 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: main() 2024-01-15T22:32:23.910 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:32:23.910 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: r = ctx.func(ctx) 2024-01-15T22:32:23.910 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:32:23.910 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:32:23.910 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:32:23.911 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:32:23.911 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:32:23.911 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:32:23.911 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83 2024-01-15T22:32:23.911 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: f90d2", line 1927, in _validate_fsid 2024-01-15T22:32:23.911 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:32:23.911 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:32:23.911 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:32:23.911 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:32:23.911 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:32:23.911 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:23 smithi099 conmon[35292]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpwgq2u0iu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp2qfhqzgr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:32:24.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: cephadm 2024-01-15T22:32:22.528679+0000 mgr.smithi099.cmjeyn (mgr.14184) 190 : cephadm [ERR] 2024-01-15T22:32:24.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi162', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-15T22:32:24.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:32:24.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:32:24.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: return f(*arg) 2024-01-15T22:32:24.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:32:24.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:32:24.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:32:24.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: code, '\n'.join(err))) 2024-01-15T22:32:24.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:32:24.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpwgq2u0iu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp2qfhqzgr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:32:24.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:32:24.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:32:24.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:32:24.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new e8a59035-6972-4653-bd6d-383fb2963762 2024-01-15T22:32:24.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:32:24.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:32:24.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:32:24.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:32:24.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:32:24.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr self.prepare() 2024-01-15T22:32:24.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:32:24.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:32:24.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:32:24.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:32:24.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:32:24.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:32:24.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:32:24.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:32:24.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:32:24.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:32:24.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:32:24.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:32:24.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:32:24.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:32:24.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:32:24.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:32:24.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:32:24.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:32:24.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:32:24.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:32:24.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:32:24.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:32:24.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:32:24.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:32:24.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:32:24.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:32:24.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:32:24.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:32:24.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:32:24.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:32:24.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:32:24.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:32:24.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:32:24.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:32:24.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:32:24.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:32:24.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:32:24.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:32:24.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:32:24.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:32:24.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:32:24.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:32:24.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:32:24.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:32:24.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:32:24.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:32:24.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:32:24.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:32:24.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:32:24.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:32:24.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:32:24.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:32:24.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: main() 2024-01-15T22:32:24.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:32:24.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: r = ctx.func(ctx) 2024-01-15T22:32:24.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:32:24.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:32:24.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:32:24.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:32:24.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:32:24.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:32:24.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 192 2024-01-15T22:32:24.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: 7, in _validate_fsid 2024-01-15T22:32:24.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:32:24.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:32:24.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:32:24.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:32:24.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:32:24.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:23 smithi162 conmon[39755]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpwgq2u0iu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp2qfhqzgr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:32:25.061 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: cluster 2024-01-15T22:32:22.939204+0000 mgr.smithi099.cmjeyn (mgr.14184) 191 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:25.061 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: cephadm 2024-01-15T22:32:23.043394+0000 mgr.smithi099.cmjeyn (mgr.14184) 192 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi162', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-15T22:32:25.061 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:32:25.061 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:32:25.061 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: return f(*arg) 2024-01-15T22:32:25.061 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:32:25.061 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:32:25.061 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:32:25.061 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: code, '\n'.join(err))) 2024-01-15T22:32:25.061 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi099/config 2024-01-15T22:32:25.061 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp9c0pn0fq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpn16n3owz:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:32:25.061 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:32:25.061 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:32:25.062 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:32:25.062 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new ea0d0325-0bb3-4cdd-ae49-9716f1cfb1cc 2024-01-15T22:32:25.062 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:32:25.062 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:32:25.062 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:32:25.062 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:32:25.062 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:32:25.062 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr self.prepare() 2024-01-15T22:32:25.062 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:32:25.062 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:32:25.062 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:32:25.062 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:32:25.062 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:32:25.062 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:32:25.063 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:32:25.063 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:32:25.063 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:32:25.063 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:32:25.063 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:32:25.063 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:32:25.063 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:32:25.063 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:32:25.063 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:32:25.063 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:32:25.063 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:32:25.063 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:32:25.063 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:32:25.063 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:32:25.063 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:32:25.064 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:32:25.064 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:32:25.064 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:32:25.064 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:32:25.064 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:32:25.064 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:32:25.064 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:32:25.064 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:32:25.064 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:32:25.064 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:32:25.064 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:32:25.064 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:32:25.064 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:32:25.064 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:32:25.065 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:32:25.065 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:32:25.065 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:32:25.065 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:32:25.065 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:32:25.065 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:32:25.065 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:32:25.065 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:32:25.065 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:32:25.065 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:32:25.065 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:32:25.065 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:32:25.065 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:32:25.065 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:32:25.065 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:32:25.066 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:32:25.066 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:32:25.066 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: main() 2024-01-15T22:32:25.066 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:32:25.066 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: r = ctx.func(ctx) 2024-01-15T22:32:25.066 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:32:25.066 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:32:25.066 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:32:25.066 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:32:25.066 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2 2024-01-15T22:32:25.066 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: 645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:32:25.066 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:32:25.066 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:32:25.066 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:32:25.066 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:32:25.067 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:32:25.067 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:32:25.067 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:32:25.067 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp9c0pn0fq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpn16n3owz:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:32:25.067 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: cephadm 2024-01-15T22:32:23.044485+0000 mgr.smithi099.cmjeyn (mgr.14184) 193 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-15T22:32:25.067 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: service_id: all-available-devices 2024-01-15T22:32:25.067 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: service_name: osd.all-available-devices 2024-01-15T22:32:25.067 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: placement: 2024-01-15T22:32:25.067 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: host_pattern: '*' 2024-01-15T22:32:25.067 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: spec: 2024-01-15T22:32:25.067 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: data_devices: 2024-01-15T22:32:25.067 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: all: true 2024-01-15T22:32:25.067 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: filter_logic: AND 2024-01-15T22:32:25.067 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: objectstore: bluestore 2024-01-15T22:32:25.067 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:32:25.067 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpwgq2u0iu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp2qfhqzgr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:32:25.068 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:32:25.068 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:32:25.068 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:32:25.068 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new e8a59035-6972-4653-bd6d-383fb2963762 2024-01-15T22:32:25.068 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:32:25.068 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:32:25.068 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:32:25.068 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:32:25.068 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:32:25.068 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr self.prepare() 2024-01-15T22:32:25.068 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:32:25.068 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:32:25.068 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:32:25.068 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:32:25.069 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:32:25.069 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:32:25.069 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:32:25.069 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:32:25.069 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:32:25.069 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:32:25.069 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:32:25.069 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:32:25.069 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:32:25.069 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:32:25.069 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:32:25.069 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:32:25.069 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:32:25.069 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:32:25.069 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:32:25.069 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:32:25.070 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:32:25.070 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:32:25.070 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:32:25.070 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:32:25.070 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:32:25.070 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:32:25.070 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:32:25.070 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:32:25.070 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:32:25.070 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:32:25.070 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:32:25.070 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:32:25.070 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:32:25.070 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:32:25.070 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:32:25.070 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:32:25.071 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:32:25.071 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:32:25.071 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:32:25.071 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:32:25.071 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:32:25.071 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:32:25.071 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:32:25.071 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py 2024-01-15T22:32:25.071 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: ", line 16, in is_root 2024-01-15T22:32:25.071 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:32:25.071 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:32:25.071 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:32:25.071 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:32:25.071 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:32:25.071 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:32:25.071 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:32:25.072 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:32:25.072 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: main() 2024-01-15T22:32:25.072 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:32:25.072 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: r = ctx.func(ctx) 2024-01-15T22:32:25.072 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:32:25.072 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:32:25.072 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:32:25.072 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:32:25.072 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:32:25.072 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:32:25.072 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:32:25.072 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:32:25.072 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:32:25.072 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:32:25.073 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:32:25.073 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:32:25.073 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpwgq2u0iu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp2qfhqzgr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:32:25.073 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:32:25.073 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-15T22:32:25.073 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: if self._apply_service(spec): 2024-01-15T22:32:25.073 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-15T22:32:25.073 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-15T22:32:25.073 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-15T22:32:25.073 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-15T22:32:25.073 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-15T22:32:25.073 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-15T22:32:25.073 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-15T22:32:25.073 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-15T22:32:25.073 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-15T22:32:25.074 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: raise self._value 2024-01-15T22:32:25.074 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-15T22:32:25.074 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: result = (True, func(*args, **kwds)) 2024-01-15T22:32:25.074 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-15T22:32:25.074 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: return list(map(*args)) 2024-01-15T22:32:25.074 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:32:25.074 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: return f(*arg) 2024-01-15T22:32:25.074 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:32:25.074 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:32:25.074 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:32:25.074 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: code, '\n'.join(err))) 2024-01-15T22:32:25.074 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:32:25.074 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpwgq2u0iu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp2qfhqzgr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:32:25.074 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:32:25.074 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:32:25.074 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:32:25.075 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new e8a59035-6972-4653-bd6d-383fb2963762 2024-01-15T22:32:25.075 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:32:25.075 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:32:25.075 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:32:25.075 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:32:25.075 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:32:25.075 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr self.prepare() 2024-01-15T22:32:25.075 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:32:25.075 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:32:25.075 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:32:25.075 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:32:25.075 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:32:25.075 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:32:25.075 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:32:25.075 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:32:25.076 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:32:25.076 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:32:25.076 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:32:25.076 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:32:25.076 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:32:25.076 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:32:25.076 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:32:25.076 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:32:25.076 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:32:25.076 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:32:25.076 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:32:25.076 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:32:25.076 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:32:25.076 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:32:25.076 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:32:25.077 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:32:25.077 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:32:25.077 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:32:25.077 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:32:25.077 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:32:25.077 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:32:25.077 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:32:25.077 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:32:25.077 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:32:25.077 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:32:25.077 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:32:25.077 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:32:25.077 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:32:25.077 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:32:25.077 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:32:25.077 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:32:25.078 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:32:25.078 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:32:25.078 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:32:25.078 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:32:25.078 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:32:25.078 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:32:25.078 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:32:25.078 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:32:25.078 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:32:25.078 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:32:25.078 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:32:25.078 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:32:25.078 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:32:25.078 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:32:25.078 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: main() 2024-01-15T22:32:25.079 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:32:25.079 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: r = ctx.func(ctx) 2024-01-15T22:32:25.079 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:32:25.079 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:32:25.079 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:32:25.079 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:32:25.079 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:32:25.079 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:32:25.079 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:32:25.079 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:32:25.079 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:32:25.079 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:32:25.079 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:32:25.079 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:32:25.079 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpwgq2u0iu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp2qfhqzgr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:32:25.079 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: cluster 2024-01-15T22:32:23.044954+0000 mgr.smithi099.cmjeyn (mgr.14184) 194 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:25.080 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:24 smithi099 conmon[35292]: cluster 2024-01-15T22:32:23.699384+0000 mon.smithi099 (mon.0) 553 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-15T22:32:25.197 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: cluster 2024-01-15T22:32:22.939204+0000 mgr.smithi099.cmjeyn (mgr.14184) 191 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:25.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: cephadm 2024-01-15T22:32:23.043394+0000 mgr.smithi099.cmjeyn (mgr.14184) 192 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi162', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-15T22:32:25.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:32:25.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:32:25.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: return f(*arg) 2024-01-15T22:32:25.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:32:25.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:32:25.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:32:25.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: code, '\n'.join(err))) 2024-01-15T22:32:25.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi099/config 2024-01-15T22:32:25.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp9c0pn0fq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpn16n3owz:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:32:25.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:32:25.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:32:25.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:32:25.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new ea0d0325-0bb3-4cdd-ae49-9716f1cfb1cc 2024-01-15T22:32:25.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:32:25.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:32:25.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:32:25.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:32:25.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:32:25.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr self.prepare() 2024-01-15T22:32:25.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:32:25.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:32:25.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:32:25.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:32:25.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:32:25.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:32:25.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:32:25.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:32:25.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:32:25.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:32:25.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:32:25.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:32:25.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:32:25.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:32:25.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:32:25.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:32:25.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:32:25.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:32:25.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:32:25.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:32:25.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:32:25.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:32:25.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:32:25.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:32:25.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:32:25.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:32:25.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:32:25.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:32:25.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:32:25.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:32:25.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:32:25.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:32:25.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:32:25.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:32:25.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:32:25.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:32:25.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:32:25.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:32:25.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:32:25.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:32:25.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:32:25.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:32:25.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:32:25.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:32:25.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:32:25.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:32:25.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:32:25.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:32:25.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:32:25.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:32:25.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:32:25.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:32:25.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: main() 2024-01-15T22:32:25.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:32:25.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: r = ctx.func(ctx) 2024-01-15T22:32:25.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:32:25.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:32:25.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:32:25.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:32:25.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:32:25.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:32:25.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-8777 2024-01-15T22:32:25.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: 4f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:32:25.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:32:25.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:32:25.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:32:25.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:32:25.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:32:25.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp9c0pn0fq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpn16n3owz:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:32:25.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: cephadm 2024-01-15T22:32:23.044485+0000 mgr.smithi099.cmjeyn (mgr.14184) 193 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-15T22:32:25.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: service_id: all-available-devices 2024-01-15T22:32:25.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: service_name: osd.all-available-devices 2024-01-15T22:32:25.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: placement: 2024-01-15T22:32:25.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: host_pattern: '*' 2024-01-15T22:32:25.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: spec: 2024-01-15T22:32:25.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: data_devices: 2024-01-15T22:32:25.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: all: true 2024-01-15T22:32:25.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: filter_logic: AND 2024-01-15T22:32:25.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: objectstore: bluestore 2024-01-15T22:32:25.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:32:25.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpwgq2u0iu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp2qfhqzgr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:32:25.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:32:25.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:32:25.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:32:25.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new e8a59035-6972-4653-bd6d-383fb2963762 2024-01-15T22:32:25.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:32:25.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:32:25.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:32:25.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:32:25.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:32:25.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr self.prepare() 2024-01-15T22:32:25.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:32:25.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:32:25.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:32:25.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:32:25.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:32:25.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:32:25.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:32:25.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:32:25.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:32:25.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:32:25.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:32:25.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:32:25.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:32:25.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:32:25.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:32:25.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:32:25.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:32:25.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:32:25.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:32:25.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:32:25.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:32:25.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:32:25.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:32:25.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:32:25.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:32:25.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:32:25.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:32:25.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:32:25.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:32:25.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:32:25.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:32:25.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:32:25.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:32:25.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:32:25.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:32:25.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:32:25.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:32:25.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:32:25.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:32:25.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:32:25.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:32:25.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:32:25.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:32:25.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:32:25.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:32:25.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/ 2024-01-15T22:32:25.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: lvm/zap.py", line 302, in zap_osd 2024-01-15T22:32:25.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:32:25.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:32:25.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:32:25.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:32:25.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:32:25.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:32:25.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: main() 2024-01-15T22:32:25.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:32:25.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: r = ctx.func(ctx) 2024-01-15T22:32:25.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:32:25.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:32:25.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:32:25.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:32:25.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:32:25.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:32:25.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:32:25.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:32:25.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:32:25.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:32:25.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:32:25.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:32:25.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpwgq2u0iu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp2qfhqzgr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:32:25.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:32:25.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-15T22:32:25.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: if self._apply_service(spec): 2024-01-15T22:32:25.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-15T22:32:25.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-15T22:32:25.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-15T22:32:25.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-15T22:32:25.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-15T22:32:25.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-15T22:32:25.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-15T22:32:25.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-15T22:32:25.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-15T22:32:25.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: raise self._value 2024-01-15T22:32:25.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-15T22:32:25.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: result = (True, func(*args, **kwds)) 2024-01-15T22:32:25.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-15T22:32:25.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: return list(map(*args)) 2024-01-15T22:32:25.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:32:25.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: return f(*arg) 2024-01-15T22:32:25.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:32:25.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:32:25.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:32:25.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: code, '\n'.join(err))) 2024-01-15T22:32:25.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:32:25.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpwgq2u0iu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp2qfhqzgr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:32:25.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:32:25.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:32:25.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:32:25.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new e8a59035-6972-4653-bd6d-383fb2963762 2024-01-15T22:32:25.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:32:25.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:32:25.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:32:25.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:32:25.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:32:25.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr self.prepare() 2024-01-15T22:32:25.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:32:25.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:32:25.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:32:25.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:32:25.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:32:25.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:32:25.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:32:25.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:32:25.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:32:25.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:32:25.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:32:25.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:32:25.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:32:25.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:32:25.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:32:25.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:32:25.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:32:25.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:32:25.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:32:25.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:32:25.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:32:25.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:32:25.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:32:25.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:32:25.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:32:25.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:32:25.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:32:25.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:32:25.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:32:25.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:32:25.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:32:25.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:32:25.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:32:25.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:32:25.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:32:25.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:32:25.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:32:25.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:32:25.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:32:25.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:32:25.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:32:25.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:32:25.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:32:25.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:32:25.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:32:25.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:32:25.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:32:25.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:32:25.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:32:25.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:32:25.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:32:25.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:32:25.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:32:25.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: main() 2024-01-15T22:32:25.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:32:25.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: r = ctx.func(ctx) 2024-01-15T22:32:25.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:32:25.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:32:25.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:32:25.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:32:25.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:32:25.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:32:25.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:32:25.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:32:25.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:32:25.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:32:25.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:32:25.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:32:25.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpwgq2u0iu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp2qfhqzgr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:32:25.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: cluster 2024-01-15T22:32:23.044954+0000 mgr.smithi099.cmjeyn (mgr.14184) 194 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:25.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:24 smithi162 conmon[39755]: cluster 2024-01-15T22:32:23.699384+0000 mon.smithi099 (mon.0) 553 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-15T22:32:25.407 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:32:26.005 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:32:26.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:25 smithi099 conmon[35292]: cluster 2024-01-15T22:32:25.045382+0000 mgr.smithi099.cmjeyn (mgr.14184) 195 : cluster 2024-01-15T22:32:26.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:25 smithi099 conmon[35292]: [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:26.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:25 smithi099 conmon[35292]: audit 2024-01-15T22:32:25.407324+0000 mon.smithi099 (mon.0) 554 : audit [DBG] from='client.? 172.21.15.99:0/172913210' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:32:26.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:25 smithi162 conmon[39755]: cluster 2024-01-15T22:32:25.045382+0000 mgr.smithi099.cmjeyn (mgr.14184) 195 : cluster 2024-01-15T22:32:26.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:25 smithi162 conmon[39755]: [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:26.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:25 smithi162 conmon[39755]: audit 2024-01-15T22:32:25.407324+0000 mon.smithi099 (mon.0) 554 : audit [DBG] from='client.? 172.21.15.99:0/172913210' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:32:27.006 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:32:28.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:28 smithi099 conmon[35292]: cluster 2024-01-15T22:32:27.045816+0000 mgr.smithi099.cmjeyn (mgr.14184) 196 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:28.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:28 smithi162 conmon[39755]: cluster 2024-01-15T22:32:27.045816+0000 mgr.smithi099.cmjeyn (mgr.14184) 196 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:28.753 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:32:29.353 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:32:29.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:29 smithi099 conmon[35292]: audit 2024-01-15T22:32:28.752606+0000 mon.smithi099 (mon.0) 555 : audit 2024-01-15T22:32:29.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:29 smithi099 conmon[35292]: [DBG] from='client.? 172.21.15.99:0/3468850325' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:32:29.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:29 smithi162 conmon[39755]: audit 2024-01-15T22:32:28.752606+0000 mon.smithi099 (mon.0) 555 : audit [DBG] from='client.? 172.21.15.99:0/3468850325' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:32:30.353 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:32:30.386 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:30 smithi099 conmon[35292]: cluster 2024-01-15T22:32:29.046224+0000 mgr.smithi099.cmjeyn (mgr.14184) 197 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:30.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:30 smithi162 conmon[39755]: cluster 2024-01-15T22:32:29.046224+0000 mgr.smithi099.cmjeyn (mgr.14184) 197 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:32.114 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:32:32.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:32 smithi099 conmon[35292]: cluster 2024-01-15T22:32:31.046854+0000 mgr.smithi099.cmjeyn (mgr.14184) 198 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:32.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:32 smithi162 conmon[39755]: cluster 2024-01-15T22:32:31.046854+0000 mgr.smithi099.cmjeyn (mgr.14184) 198 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:32.719 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:32:33.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:33 smithi099 conmon[35292]: audit 2024-01-15T22:32:32.113630+0000 mon.smithi099 (mon.0) 556 : audit [DBG] 2024-01-15T22:32:33.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:33 smithi099 conmon[35292]: from='client.? 172.21.15.99:0/4191319978' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:32:33.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:33 smithi162 conmon[39755]: audit 2024-01-15T22:32:32.113630+0000 mon.smithi099 (mon.0) 556 : audit [DBG] from='client.? 172.21.15.99:0/4191319978' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:32:33.720 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:32:34.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:34 smithi099 conmon[35292]: cluster 2024-01-15T22:32:33.047422+0000 mgr.smithi099.cmjeyn (mgr.14184) 199 : cluster [DBG] 2024-01-15T22:32:34.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:34 smithi099 conmon[35292]: pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:34.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:34 smithi162 conmon[39755]: cluster 2024-01-15T22:32:33.047422+0000 mgr.smithi099.cmjeyn (mgr.14184) 199 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:35.551 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:32:36.116 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:32:36.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:35 smithi099 conmon[35292]: cluster 2024-01-15T22:32:35.047897+0000 mgr.smithi099.cmjeyn (mgr.14184) 200 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:36.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:35 smithi099 conmon[35292]: audit 2024-01-15T22:32:35.551023+0000 mon.smithi099 (mon.0) 557 : audit [DBG] from='client.? 172.21.15.99:0/3398386441' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:32:36.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:35 smithi162 conmon[39755]: cluster 2024-01-15T22:32:35.047897+0000 mgr.smithi099.cmjeyn (mgr.14184) 200 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:36.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:35 smithi162 conmon[39755]: audit 2024-01-15T22:32:36.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:35 smithi162 conmon[39755]: 2024-01-15T22:32:35.551023+0000 mon.smithi099 (mon.0) 557 : audit [DBG] from='client.? 172.21.15.99:0/3398386441' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:32:37.117 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:32:38.293 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:38 smithi099 conmon[35292]: cluster 2024-01-15T22:32:37.048366+0000 mgr.smithi099.cmjeyn (mgr.14184) 201 : cluster 2024-01-15T22:32:38.293 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:38 smithi099 conmon[35292]: [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:38.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:38 smithi162 conmon[39755]: cluster 2024-01-15T22:32:37.048366+0000 mgr.smithi099.cmjeyn (mgr.14184) 201 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:38.844 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:32:39.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:39 smithi099 conmon[35292]: audit 2024-01-15T22:32:38.843573+0000 mon.smithi099 (mon.0) 558 : audit [DBG] from='client.? 172.21.15.99:0/3062281260' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:32:39.410 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:32:39.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:39 smithi162 conmon[39755]: audit 2024-01-15T22:32:38.843573+0000 mon.smithi099 (mon.0) 558 : audit [DBG] from='client.? 172.21.15.99:0/3062281260' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:32:40.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:40 smithi099 conmon[35292]: cluster 2024-01-15T22:32:39.048833+0000 mgr.smithi099.cmjeyn (mgr.14184) 202 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:40.411 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:32:40.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:40 smithi162 conmon[39755]: cluster 2024-01-15T22:32:39.048833+0000 mgr.smithi099.cmjeyn (mgr.14184) 202 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:42.231 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:32:42.393 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:42 smithi099 conmon[35292]: cluster 2024-01-15T22:32:41.049486+0000 mgr.smithi099.cmjeyn (mgr.14184) 203 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:42.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:42 smithi162 conmon[39755]: cluster 2024-01-15T22:32:41.049486+0000 mgr.smithi099.cmjeyn (mgr.14184) 203 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:42.766 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:32:43.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:43 smithi099 conmon[35292]: audit 2024-01-15T22:32:42.230850+0000 mon.smithi099 (mon.0) 559 : audit [DBG] from='client.? 172.21.15.99:0/295464429' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:32:43.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:43 smithi162 conmon[39755]: audit 2024-01-15T22:32:42.230850+0000 mon.smithi099 (mon.0) 559 : audit [DBG] from='client.? 172.21.15.99:0/295464429' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:32:43.767 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:32:44.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:44 smithi099 conmon[35292]: cluster 2024-01-15T22:32:43.049926+0000 mgr.smithi099.cmjeyn (mgr.14184) 204 : 2024-01-15T22:32:44.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:44 smithi099 conmon[35292]: cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:44.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:44 smithi162 conmon[39755]: cluster 2024-01-15T22:32:43.049926+0000 mgr.smithi099.cmjeyn (mgr.14184) 204 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:45.512 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:32:46.082 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:32:46.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:45 smithi099 conmon[35292]: cluster 2024-01-15T22:32:45.050308+0000 mgr.smithi099.cmjeyn (mgr.14184) 205 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:46.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:45 smithi099 conmon[35292]: audit 2024-01-15T22:32:45.511999+0000 mon.smithi099 (mon.0) 560 : audit [DBG] from='client.? 172.21.15.99:0/478012042' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:32:46.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:45 smithi162 conmon[39755]: cluster 2024-01-15T22:32:45.050308+0000 mgr.smithi099.cmjeyn (mgr.14184) 205 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:46.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:45 smithi162 conmon[39755]: audit 2024-01-15T22:32:45.511999+0000 mon.smithi099 (mon.0) 560 : audit [DBG] from='client.? 172.21.15.99:0/478012042' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:32:47.083 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:32:48.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:48 smithi099 conmon[35292]: cluster 2024-01-15T22:32:47.050879+0000 mgr.smithi099.cmjeyn (mgr.14184) 206 2024-01-15T22:32:48.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:48 smithi099 conmon[35292]: : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:48.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:48 smithi162 conmon[39755]: cluster 2024-01-15T22:32:47.050879+0000 mgr.smithi099.cmjeyn (mgr.14184) 206 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:48.834 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:32:49.370 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:49 smithi099 conmon[35292]: audit 2024-01-15T22:32:48.833061+0000 mon.smithi099 (mon.0) 561 : audit [DBG] from='client.? 172.21.15.99:0/3774249207' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:32:49.372 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:32:49.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:49 smithi162 conmon[39755]: audit 2024-01-15T22:32:48.833061+0000 mon.smithi099 (mon.0) 561 : audit [DBG] from='client.? 172.21.15.99:0/3774249207' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:32:50.372 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:32:50.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:50 smithi099 conmon[35292]: cluster 2024-01-15T22:32:49.051329+0000 mgr.smithi099.cmjeyn (mgr.14184) 207 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:50.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:50 smithi162 conmon[39755]: cluster 2024-01-15T22:32:49.051329+0000 mgr.smithi099.cmjeyn (mgr.14184) 207 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:52.339 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:32:52.360 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:52 smithi099 conmon[35292]: cluster 2024-01-15T22:32:51.051999+0000 mgr.smithi099.cmjeyn (mgr.14184) 208 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:52.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:52 smithi162 conmon[39755]: cluster 2024-01-15T22:32:51.051999+0000 mgr.smithi099.cmjeyn (mgr.14184) 208 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:52.888 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:32:53.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:53 smithi099 conmon[35292]: audit 2024-01-15T22:32:52.338199+0000 mon.smithi099 (mon.0) 562 : audit 2024-01-15T22:32:53.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:53 smithi099 conmon[35292]: [DBG] from='client.? 172.21.15.99:0/1847178832' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:32:53.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:53 smithi162 conmon[39755]: audit 2024-01-15T22:32:52.338199+0000 mon.smithi099 (mon.0) 562 : audit [DBG] from='client.? 172.21.15.99:0/1847178832' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:32:53.889 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:32:54.347 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:54 smithi099 conmon[35292]: cluster 2024-01-15T22:32:53.052474+0000 mgr.smithi099.cmjeyn (mgr.14184) 209 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:54.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:54 smithi162 conmon[39755]: cluster 2024-01-15T22:32:53.052474+0000 mgr.smithi099.cmjeyn (mgr.14184) 209 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:55.705 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:32:56.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:55 smithi099 conmon[35292]: cluster 2024-01-15T22:32:55.052904+0000 mgr.smithi099.cmjeyn (mgr.14184) 210 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:56.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:55 smithi099 conmon[35292]: audit 2024-01-15T22:32:55.705051+0000 mon.smithi099 (mon.0) 563 : audit [DBG] from='client.? 172.21.15.99:0/3559183469' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:32:56.212 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:32:56.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:55 smithi162 conmon[39755]: cluster 2024-01-15T22:32:55.052904+0000 mgr.smithi099.cmjeyn (mgr.14184) 210 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:56.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:55 smithi162 conmon[39755]: audit 2024-01-15T22:32:55.705051+0000 mon.smithi099 (mon.0) 563 : audit [DBG] from='client.? 172.21.15.99:0/3559183469' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:32:57.213 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:32:58.397 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:58 smithi099 conmon[35292]: cluster 2024-01-15T22:32:57.053456+0000 mgr.smithi099.cmjeyn (mgr.14184 2024-01-15T22:32:58.398 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:58 smithi099 conmon[35292]: ) 211 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:58.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:58 smithi162 conmon[39755]: cluster 2024-01-15T22:32:57.053456+0000 mgr.smithi099.cmjeyn (mgr.14184) 211 : cluster [DBG] 2024-01-15T22:32:58.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:58 smithi162 conmon[39755]: pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:32:58.924 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:32:59.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:32:59 smithi099 conmon[35292]: audit 2024-01-15T22:32:58.923614+0000 mon.smithi099 (mon.0) 564 : audit [DBG] from='client.? 172.21.15.99:0/528799586' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:32:59.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:32:59 smithi162 conmon[39755]: audit 2024-01-15T22:32:58.923614+0000 mon.smithi099 (mon.0) 564 : audit [DBG] from='client.? 172.21.15.99:0/528799586' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:32:59.487 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:33:00.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:00 smithi099 conmon[35292]: cluster 2024-01-15T22:32:59.053939+0000 mgr.smithi099.cmjeyn (mgr.14184) 212 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:00.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:00 smithi162 conmon[39755]: cluster 2024-01-15T22:32:59.053939+0000 mgr.smithi099.cmjeyn (mgr.14184) 212 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:00.488 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:33:02.289 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:33:02.311 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:02 smithi099 conmon[35292]: cluster 2024-01-15T22:33:01.054320+0000 mgr.smithi099.cmjeyn (mgr.14184) 213 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:02.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:02 smithi162 conmon[39755]: cluster 2024-01-15T22:33:01.054320+0000 mgr.smithi099.cmjeyn (mgr.14184) 213 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:02.836 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:33:03.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:03 smithi099 conmon[35292]: audit 2024-01-15T22:33:02.288839+0000 2024-01-15T22:33:03.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:03 smithi099 conmon[35292]: mon.smithi099 (mon.0) 565 : audit [DBG] from='client.? 172.21.15.99:0/2230903645' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:33:03.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:03 smithi162 conmon[39755]: audit 2024-01-15T22:33:02.288839+0000 mon.smithi099 (mon.0) 565 : audit [DBG] from='client.? 172.21.15.99:0/2230903645' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:33:03.838 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:33:04.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:04 smithi099 conmon[35292]: cluster 2024-01-15T22:33:03.054705+0000 mgr.smithi099.cmjeyn (mgr.14184) 214 : cluster 2024-01-15T22:33:04.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:04 smithi099 conmon[35292]: [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:04.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:04 smithi162 conmon[39755]: cluster 2024-01-15T22:33:03.054705+0000 mgr.smithi099.cmjeyn (mgr.14184) 214 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:05.692 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:33:06.336 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:05 smithi099 conmon[35292]: cluster 2024-01-15T22:33:05.055283+0000 mgr.smithi099.cmjeyn (mgr.14184) 215 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:06.337 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:05 smithi099 conmon[35292]: audit 2024-01-15T22:33:05.691038+0000 mon.smithi099 (mon.0) 566 : audit [DBG] from='client.? 172.21.15.99:0/2781089045' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:33:06.354 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:33:06.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:05 smithi162 conmon[39755]: cluster 2024-01-15T22:33:05.055283+0000 mgr.smithi099.cmjeyn (mgr.14184) 215 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:06.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:05 smithi162 conmon[39755]: audit 2024-01-15T22:33:05.691038+0000 mon.smithi099 (mon.0) 566 : audit [DBG] from='client.? 172.21.15.99:0/2781089045' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:33:07.355 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:33:08.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:08 smithi099 conmon[35292]: cluster 2024-01-15T22:33:07.055725+0000 2024-01-15T22:33:08.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:08 smithi099 conmon[35292]: mgr.smithi099.cmjeyn (mgr.14184) 216 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:08.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:08 smithi162 conmon[39755]: cluster 2024-01-15T22:33:07.055725+0000 mgr.smithi099.cmjeyn (mgr.14184) 216 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:08.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:08 smithi162 conmon[39755]: 2024-01-15T22:33:09.063 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:33:09.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:09 smithi099 conmon[35292]: audit 2024-01-15T22:33:09.062015+0000 mon.smithi099 (mon.0) 567 : audit [DBG] from='client.? 172.21.15.99:0/2732504876' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:33:09.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:09 smithi162 conmon[39755]: audit 2024-01-15T22:33:09.062015+0000 mon.smithi099 (mon.0) 567 : audit [DBG] from='client.? 172.21.15.99:0/2732504876' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:33:09.651 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:33:10.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:10 smithi099 conmon[35292]: cluster 2024-01-15T22:33:09.056160+0000 mgr.smithi099.cmjeyn (mgr.14184) 217 : cluster 2024-01-15T22:33:10.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:10 smithi099 conmon[35292]: [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:10.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:10 smithi162 conmon[39755]: cluster 2024-01-15T22:33:09.056160+0000 mgr.smithi099.cmjeyn (mgr.14184) 217 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:10.652 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:33:12.386 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:33:12.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:12 smithi099 conmon[35292]: cluster 2024-01-15T22:33:11.056576+0000 mgr.smithi099.cmjeyn (mgr.14184) 218 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:12.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:12 smithi162 conmon[39755]: cluster 2024-01-15T22:33:11.056576+0000 mgr.smithi099.cmjeyn (mgr.14184) 218 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:12.910 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:33:13.154 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:13 smithi099 conmon[35292]: audit 2024-01-15T22:33:12.385132+0000 mon.smithi099 (mon.0) 568 : audit [DBG] 2024-01-15T22:33:13.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:13 smithi099 conmon[35292]: from='client.? 172.21.15.99:0/2774291429' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:33:13.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:13 smithi162 conmon[39755]: audit 2024-01-15T22:33:12.385132+0000 mon.smithi099 (mon.0) 568 : audit [DBG] from='client.? 172.21.15.99:0/2774291429' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:33:13.912 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:33:14.352 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:14 smithi099 conmon[35292]: cluster 2024-01-15T22:33:13.057097+0000 mgr.smithi099.cmjeyn (mgr.14184) 219 : cluster [DBG] 2024-01-15T22:33:14.352 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:14 smithi099 conmon[35292]: pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:14.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:14 smithi162 conmon[39755]: cluster 2024-01-15T22:33:13.057097+0000 mgr.smithi099.cmjeyn (mgr.14184) 219 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:15.664 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:33:16.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:15 smithi099 conmon[35292]: cluster 2024-01-15T22:33:15.057517+0000 mgr.smithi099.cmjeyn (mgr.14184) 220 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:16.243 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:15 smithi099 conmon[35292]: audit 2024-01-15T22:33:15.663795+0000 mon.smithi099 (mon.0) 569 : audit [DBG] from='client.? 172.21.15.99:0/3007364991' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:33:16.258 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:33:16.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:15 smithi162 conmon[39755]: cluster 2024-01-15T22:33:15.057517+0000 mgr.smithi099.cmjeyn (mgr.14184) 220 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:16.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:15 smithi162 conmon[39755]: audit 2024-01-15T22:33:15.663795+0000 mon.smithi099 (mon.0) 569 2024-01-15T22:33:16.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:15 smithi162 conmon[39755]: : audit [DBG] from='client.? 172.21.15.99:0/3007364991' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:33:17.259 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:33:18.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:18 smithi099 conmon[35292]: cluster 2024-01-15T22:33:17.057975+0000 mgr.smithi099.cmjeyn (mgr.14184) 221 : cluster 2024-01-15T22:33:18.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:18 smithi099 conmon[35292]: [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:18.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:18 smithi162 conmon[39755]: cluster 2024-01-15T22:33:17.057975+0000 mgr.smithi099.cmjeyn (mgr.14184) 221 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:18.995 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:33:19.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:19 smithi099 conmon[35292]: audit 2024-01-15T22:33:18.994445+0000 mon.smithi099 (mon.0) 570 : audit [DBG] from='client.? 172.21.15.99:0/1348109354' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:33:19.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:19 smithi162 conmon[39755]: audit 2024-01-15T22:33:18.994445+0000 mon.smithi099 (mon.0) 570 : audit [DBG] from='client.? 172.21.15.99:0/1348109354' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:33:19.565 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:33:20.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:20 smithi099 conmon[35292]: cluster 2024-01-15T22:33:19.058363+0000 mgr.smithi099.cmjeyn (mgr.14184) 222 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:20.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:20 smithi162 conmon[39755]: cluster 2024-01-15T22:33:19.058363+0000 mgr.smithi099.cmjeyn (mgr.14184) 222 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:20.566 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:33:22.350 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:33:22.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:22 smithi099 conmon[35292]: cluster 2024-01-15T22:33:21.058758+0000 mgr.smithi099.cmjeyn (mgr.14184) 223 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:22.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:22 smithi162 conmon[39755]: cluster 2024-01-15T22:33:21.058758+0000 mgr.smithi099.cmjeyn (mgr.14184) 223 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:22.983 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:33:23.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:23 smithi099 conmon[35292]: audit 2024-01-15T22:33:22.350014+0000 mon.smithi099 (mon.0) 571 : audit [DBG] from='client.? 172.21.15.99:0/2856903928' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:33:23.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:23 smithi099 conmon[35292]: audit 2024-01-15T22:33:23.046039+0000 mon.smithi099 (mon.0) 572 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-15T22:33:23.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:23 smithi162 conmon[39755]: audit 2024-01-15T22:33:22.350014+0000 mon.smithi099 (mon.0) 571 : audit [DBG] from='client.? 172.21.15.99:0/2856903928' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:33:23.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:23 smithi162 conmon[39755]: audit 2024-01-15T22:33:23.046039+0000 mon.smithi099 (mon.0) 572 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-15T22:33:23.985 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:33:24.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:24 smithi099 conmon[35292]: cluster 2024-01-15T22:33:23.059071+0000 mgr.smithi099.cmjeyn (mgr.14184 2024-01-15T22:33:24.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:24 smithi099 conmon[35292]: ) 224 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:24.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:24 smithi099 conmon[35292]: audit 2024-01-15T22:33:23.368911+0000 mon.smithi099 (mon.0) 573 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:33:24.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:24 smithi099 conmon[35292]: audit 2024-01-15T22:33:23.369518+0000 mon.smithi099 (mon.0) 574 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-15T22:33:24.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:24 smithi099 conmon[35292]: audit 2024-01-15T22:33:23.373531+0000 mon.smithi099 (mon.0) 575 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:33:24.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:24 smithi099 conmon[35292]: cluster 2024-01-15T22:33:23.373829+0000 mgr.smithi099.cmjeyn (mgr.14184) 225 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:24.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:24 smithi099 conmon[35292]: audit 2024-01-15T22:33:23.376913+0000 mon.smithi099 (mon.0) 576 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-15T22:33:24.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:24 smithi099 conmon[35292]: audit 2024-01-15T22:33:23.379459+0000 mon.smithi099 (mon.0) 577 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-15T22:33:24.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:24 smithi099 conmon[35292]: audit 2024-01-15T22:33:23.380309+0000 mon.smithi099 (mon.0) 578 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-15T22:33:24.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:24 smithi099 conmon[35292]: audit 2024-01-15T22:33:23.380502+0000 mon.smithi099 (mon.0) 579 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:33:24.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:24 smithi099 conmon[35292]: audit 2024-01-15T22:33:23.380892+0000 mon.smithi099 (mon.0) 580 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:33:24.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:24 smithi162 conmon[39755]: cluster 2024-01-15T22:33:23.059071+0000 mgr.smithi099.cmjeyn (mgr.14184) 224 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:24.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:24 smithi162 conmon[39755]: audit 2024-01-15T22:33 2024-01-15T22:33:24.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:24 smithi162 conmon[39755]: :23.368911+0000 mon.smithi099 (mon.0) 573 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:33:24.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:24 smithi162 conmon[39755]: audit 2024-01-15T22:33:23.369518+0000 mon.smithi099 (mon.0) 574 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-15T22:33:24.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:24 smithi162 conmon[39755]: audit 2024-01-15T22:33:23.373531+0000 mon.smithi099 (mon.0) 575 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:33:24.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:24 smithi162 conmon[39755]: cluster 2024-01-15T22:33:23.373829+0000 mgr.smithi099.cmjeyn (mgr.14184) 225 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:24.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:24 smithi162 conmon[39755]: audit 2024-01-15T22:33:23.376913+0000 mon.smithi099 (mon.0) 576 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-15T22:33:24.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:24 smithi162 conmon[39755]: audit 2024-01-15T22:33:23.379459+0000 mon.smithi099 (mon.0) 577 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-15T22:33:24.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:24 smithi162 conmon[39755]: audit 2024-01-15T22:33:23.380309+0000 mon.smithi099 (mon.0) 578 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-15T22:33:24.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:24 smithi162 conmon[39755]: audit 2024-01-15T22:33:23.380502+0000 mon.smithi099 (mon.0) 579 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:33:24.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:24 smithi162 conmon[39755]: audit 2024-01-15T22:33:23.380892+0000 mon.smithi099 (mon.0) 580 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:33:25.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:25 smithi099 conmon[35292]: cluster 2024-01-15T22:33:24.371909+0000 mon.smithi099 (mon.0) 581 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-15T22:33:25.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:25 smithi099 conmon[35292]: cluster 2024-01-15T22:33:24.371946+0000 mon.smithi099 (mon.0) 582 : cluster [INF] Cluster is now healthy 2024-01-15T22:33:25.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:25 smithi162 conmon[39755]: cluster 2024-01-15T22:33:24.371909+0000 mon.smithi099 (mon.0) 581 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-15T22:33:25.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:25 smithi162 conmon[39755]: cluster 2024-01-15T22:33:24.371946+0000 mon.smithi099 (mon.0) 582 : cluster [INF] Cluster is now healthy 2024-01-15T22:33:26.356 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:33:26.384 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:26 smithi099 conmon[35292]: cluster 2024-01-15T22:33:25.374103+0000 mgr.smithi099.cmjeyn (mgr.14184) 226 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:26.384 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:26 smithi099 conmon[35292]: audit 2024-01-15T22:33:25.975003+0000 mon.smithi162 (mon.1) 16 : audit [INF] from='client.? 172.21.15.162:0/3483590761' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "683f5818-c5ab-40af-8d03-070567658d0e"}]: dispatch 2024-01-15T22:33:26.384 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:26 smithi099 conmon[35292]: audit 2024-01-15T22:33:25.975474+0000 mon.smithi099 (mon.0) 583 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "683f5818-c5ab-40af-8d03-070567658d0e"}]: dispatch 2024-01-15T22:33:26.384 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:26 smithi099 conmon[35292]: audit 2024-01-15T22:33:25.980945+0000 mon.smithi099 (mon.0) 584 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "683f5818-c5ab-40af-8d03-070567658d0e"}]': finished 2024-01-15T22:33:26.384 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:26 smithi099 conmon[35292]: cluster 2024-01-15T22:33:25.981020+0000 mon.smithi099 (mon.0) 585 2024-01-15T22:33:26.384 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:26 smithi099 conmon[35292]: : cluster [DBG] osdmap e30: 1 total, 0 up, 1 in 2024-01-15T22:33:26.384 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:26 smithi099 conmon[35292]: audit 2024-01-15T22:33:25.981153+0000 mon.smithi099 (mon.0) 586 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-15T22:33:26.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:26 smithi162 conmon[39755]: cluster 2024-01-15T22:33:25.374103+0000 mgr.smithi099.cmjeyn (mgr.14184) 226 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:26.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:26 smithi162 conmon[39755]: audit 2024-01-15T22:33:25.975003+0000 mon.smithi162 (mon.1) 16 : audit [INF] from='client.? 172.21.15.162:0/3483590761' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "683f5818-c5ab-40af-8d03-070567658d0e"}]: dispatch 2024-01-15T22:33:26.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:26 smithi162 conmon[39755]: audit 2024-01-15T22:33:25.975474+0000 mon.smithi099 (mon.0) 583 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "683f5818-c5ab-40af-8d03-070567658d0e"}]: dispatch 2024-01-15T22:33:26.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:26 smithi162 conmon[39755]: audit 2024-01-15T22:33:25.980945+0000 mon.smithi099 (mon.0) 584 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "683f5818-c5ab-40af-8d03-070567658d0e"}]': finished 2024-01-15T22:33:26.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:26 smithi162 conmon[39755]: cluster 2024-01-15T22:33:25.981020+0000 mon.smithi099 (mon.0) 585 : cluster [DBG] osdmap e30: 1 total, 0 up, 1 in 2024-01-15T22:33:26.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:26 smithi162 conmon[39755]: audit 2024-01-15T22:33:25.981153+0000 mon.smithi099 (mon.0) 586 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-15T22:33:26.957 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":31,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:33:27.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:27 smithi099 conmon[35292]: audit 2024-01-15T22:33:26.272526+0000 mon.smithi162 (mon.1) 2024-01-15T22:33:27.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:27 smithi099 conmon[35292]: 17 : audit [INF] from='client.? 172.21.15.162:0/583900053' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:33:27.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:27 smithi099 conmon[35292]: audit 2024-01-15T22:33:26.272972+0000 mon.smithi099 (mon.0) 587 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:33:27.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:27 smithi099 conmon[35292]: audit 2024-01-15T22: 2024-01-15T22:33:27.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:27 smithi099 conmon[35292]: 33:26.278253+0000 mon.smithi099 (mon.0) 588 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-15T22:33:27.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:27 smithi099 conmon[35292]: cluster 2024-01-15T22:33:26. 2024-01-15T22:33:27.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:27 smithi099 conmon[35292]: 278303+0000 mon.smithi099 (mon.0) 589 : cluster [DBG] osdmap e31: 0 total, 0 up, 0 in 2024-01-15T22:33:27.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:27 smithi099 conmon[35292]: audit 2024-01-15T22:33:26.355121 2024-01-15T22:33:27.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:27 smithi099 conmon[35292]: +0000 mon.smithi099 (mon.0) 590 : audit [DBG] 2024-01-15T22:33:27.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:27 smithi099 conmon[35292]: from='client.? 172.21.15.99:0/1423326231' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:33:27.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:27 smithi099 conmon[35292]: audit 2024-01- 2024-01-15T22:33:27.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:27 smithi099 conmon[35292]: 15T22:33:26.559800+0000 mon.smithi099 (mon.0) 591 : audit [INF] 2024-01-15T22:33:27.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:27 smithi099 conmon[35292]: from='client.? 172.21.15.99:0/234864822' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e7735357-e3f2-4fd3-acab-4220dcdebd43"}]: dispatch 2024-01-15T22:33:27.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:27 smithi099 conmon[35292]: audit 2024- 2024-01-15T22:33:27.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:27 smithi099 conmon[35292]: 01-15T22:33:26.563499+0000 mon.smithi099 (mon.0) 592 : audit [INF] from='client.? 172.21.15.99:0/234864822' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e7735357-e3f2-4fd3-acab-4220dcdebd43"}]': finished 2024-01-15T22:33:27.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:27 smithi099 conmon[35292]: cluster 2024-01-15T22:33:26.563553 2024-01-15T22:33:27.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:27 smithi099 conmon[35292]: +0000 mon.smithi099 (mon.0) 593 : cluster [DBG] osdmap e32: 1 total, 0 up, 1 in 2024-01-15T22:33:27.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:27 smithi099 conmon[35292]: audit 2024-01-15T22:33 2024-01-15T22:33:27.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:27 smithi099 conmon[35292]: :26.563624+0000 mon.smithi099 (mon.0) 594 : audit 2024-01-15T22:33:27.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:27 smithi099 conmon[35292]: [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-15T22:33:27.407 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:27 smithi099 conmon[35292]: audit 2024-01-15 2024-01-15T22:33:27.407 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:27 smithi099 conmon[35292]: T22:33:26.853789+0000 mon.smithi099 (mon.0) 595 : audit 2024-01-15T22:33:27.407 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:27 smithi099 conmon[35292]: [INF] from='client.? 172.21.15.99:0/4079924827' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:33:27.407 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:27 smithi099 conmon[35292]: audit 2024-01 2024-01-15T22:33:27.407 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:27 smithi099 conmon[35292]: -15T22:33:26.857567+0000 mon.smithi099 (mon.0) 596 : audit [INF] from='client.? 172.21.15.99:0/4079924827' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-15T22:33:27.407 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:27 smithi099 conmon[35292]: 2024-01-15T22:33:27.407 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:27 smithi099 conmon[35292]: cluster 2024-01 2024-01-15T22:33:27.407 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:27 smithi099 conmon[35292]: -15T22:33:26.857607+0000 mon.smithi099 (mon.0) 597 : cluster [DBG] osdmap e33: 0 total, 0 up, 0 in 2024-01-15T22:33:27.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:27 smithi162 conmon[39755]: audit 2024-01-15T22:33:26.272526+0000 mon.smithi162 (mon.1) 17 : 2024-01-15T22:33:27.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:27 smithi162 conmon[39755]: audit [INF] from='client.? 172.21.15.162:0/583900053' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:33:27.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:27 smithi162 conmon[39755]: audit 2024-01-15T22:33:26.272972+0000 mon.smithi099 (mon.0) 587 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:33:27.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:27 smithi162 conmon[39755]: audit 2024-01-15T22:33:26.278253+0000 mon.smithi099 (mon.0) 588 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-15T22:33:27.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:27 smithi162 conmon[39755]: cluster 2024-01-15T22:33:26.278303+0000 mon.smithi099 (mon.0) 589 : cluster [DBG] osdmap e31: 0 total, 0 up, 0 in 2024-01-15T22:33:27.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:27 smithi162 conmon[39755]: audit 2024-01-15T22:33:26.355121+0000 mon.smithi099 (mon.0) 590 : audit [DBG] from='client.? 172.21.15.99:0/1423326231' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:33:27.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:27 smithi162 conmon[39755]: audit 2024-01-15T22:33:26.559800+0000 mon.smithi099 (mon.0) 591 : audit [INF] from='client.? 172.21.15.99:0/234864822' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e7735357-e3f2-4fd3-acab-4220dcdebd43"}]: dispatch 2024-01-15T22:33:27.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:27 smithi162 conmon[39755]: audit 2024-01-15T22:33:26.563499+0000 mon.smithi099 ( 2024-01-15T22:33:27.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:27 smithi162 conmon[39755]: mon.0) 592 : audit [INF] from='client.? 172.21.15.99:0/234864822' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e7735357-e3f2-4fd3-acab-4220dcdebd43"}]': finished 2024-01-15T22:33:27.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:27 smithi162 conmon[39755]: cluster 2024-01-15T22: 2024-01-15T22:33:27.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:27 smithi162 conmon[39755]: 33:26.563553+0000 mon.smithi099 (mon.0) 593 : cluster [DBG] osdmap e32: 1 total, 0 up, 1 in 2024-01-15T22:33:27.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:27 smithi162 conmon[39755]: audit 2024-01-15T22:33:27.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:27 smithi162 conmon[39755]: 2024-01-15T22:33:26.563624+0000 mon.smithi099 (mon.0) 594 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-15T22:33:27.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:27 smithi162 conmon[39755]: 2024-01-15T22:33:27.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:27 smithi162 conmon[39755]: audit 2024-01-15T22:33: 2024-01-15T22:33:27.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:27 smithi162 conmon[39755]: 26.853789+0000 mon.smithi099 (mon.0) 595 : audit [INF] from='client.? 172.21.15.99:0/4079924827' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:33:27.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:27 smithi162 conmon[39755]: audit 2024-01-15T22:33:27.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:27 smithi162 conmon[39755]: 2024-01-15T22:33:26.857567+0000 mon.smithi099 (mon.0) 596 : audit [INF] from='client.? 172.21.15.99:0/4079924827' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-15T22:33:27.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:27 smithi162 conmon[39755]: cluster 2024-01-15T22:33:27.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:27 smithi162 conmon[39755]: 2024-01-15T22:33:26.857607+0000 mon.smithi099 (mon.0) 597 : cluster [DBG] osdmap e33: 0 total, 0 up, 0 in 2024-01-15T22:33:27.958 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:33:28.392 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: cephadm 2024-01-15T22:33:27.004663+0000 mgr.smithi099.cmjeyn (mgr.14184) 227 : 2024-01-15T22:33:28.392 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi162', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-15T22:33:28.392 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:33:28.392 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:33:28.392 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: return f(*arg) 2024-01-15T22:33:28.392 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:33:28.392 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:33:28.392 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:33:28.392 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: code, '\n'.join(err))) 2024-01-15T22:33:28.393 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:33:28.393 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpf44ann54:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpsdpvzxui:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:33:28.393 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:33:28.393 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:33:28.393 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:33:28.393 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 683f5818-c5ab-40af-8d03-070567658d0e 2024-01-15T22:33:28.393 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:33:28.393 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:33:28.393 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:33:28.393 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:33:28.393 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:33:28.394 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr self.prepare() 2024-01-15T22:33:28.394 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:33:28.394 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:33:28.394 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:33:28.394 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:33:28.394 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:33:28.394 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:33:28.394 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:33:28.394 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:33:28.394 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:33:28.394 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:33:28.394 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:33:28.394 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:33:28.394 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:33:28.394 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:33:28.394 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:33:28.395 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:33:28.395 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:33:28.395 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:33:28.395 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:33:28.395 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:33:28.395 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:33:28.395 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:33:28.395 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:33:28.395 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:33:28.395 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:33:28.395 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:33:28.395 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:33:28.395 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:33:28.395 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:33:28.396 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:33:28.396 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:33:28.396 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:33:28.396 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:33:28.396 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:33:28.396 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:33:28.396 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:33:28.396 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:33:28.396 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:33:28.396 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:33:28.396 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:33:28.396 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:33:28.396 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:33:28.396 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:33:28.397 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:33:28.397 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:33:28.397 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:33:28.397 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:33:28.397 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:33:28.397 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:33:28.397 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:33:28.397 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:33:28.397 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:33:28.397 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: main() 2024-01-15T22:33:28.397 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:33:28.397 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: r = ctx.func(ctx) 2024-01-15T22:33:28.397 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:33:28.397 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:33:28.398 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:33:28.398 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:33:28.398 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:33:28.398 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:33:28.398 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90 2024-01-15T22:33:28.398 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: d2", line 1927, in _validate_fsid 2024-01-15T22:33:28.398 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:33:28.398 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:33:28.398 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:33:28.398 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:33:28.398 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:33:28.398 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpf44ann54:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpsdpvzxui:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:33:28.398 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: cluster 2024-01-15T22:33:27.374312+0000 mgr.smithi099.cmjeyn (mgr.14184) 228 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:28.398 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: cephadm 2024-01-15T22:33:27.479500+0000 mgr.smithi099.cmjeyn (mgr.14184) 229 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi162', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-15T22:33:28.398 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:33:28.399 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:33:28.399 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: return f(*arg) 2024-01-15T22:33:28.399 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:33:28.399 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:33:28.399 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:33:28.399 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: code, '\n'.join(err))) 2024-01-15T22:33:28.399 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi099/config 2024-01-15T22:33:28.399 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpw30k0laj:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpair9xpj8:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:33:28.399 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:33:28.399 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:33:28.399 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:33:28.399 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new e7735357-e3f2-4fd3-acab-4220dcdebd43 2024-01-15T22:33:28.399 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:33:28.399 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:33:28.400 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:33:28.400 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:33:28.400 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:33:28.400 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr self.prepare() 2024-01-15T22:33:28.400 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:33:28.400 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:33:28.400 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:33:28.400 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:33:28.400 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:33:28.400 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:33:28.400 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:33:28.400 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:33:28.400 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:33:28.401 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:33:28.401 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:33:28.401 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:33:28.401 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:33:28.401 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:33:28.401 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:33:28.401 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:33:28.401 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:33:28.401 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:33:28.401 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:33:28.401 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:33:28.401 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:33:28.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:33:28.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:33:28.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:33:28.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:33:28.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:33:28.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:33:28.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:33:28.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:33:28.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:33:28.403 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:33:28.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:33:28.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:33:28.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:33:28.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:33:28.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:33:28.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:33:28.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:33:28.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:33:28.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3. 2024-01-15T22:33:28.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: 6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:33:28.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:33:28.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:33:28.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:33:28.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:33:28.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:33:28.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:33:28.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:33:28.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:33:28.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:33:28.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:33:28.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:33:28.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:33:28.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: main() 2024-01-15T22:33:28.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:33:28.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: r = ctx.func(ctx) 2024-01-15T22:33:28.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:33:28.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:33:28.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:33:28.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:33:28.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:33:28.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:33:28.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:33:28.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:33:28.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:33:28.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:33:28.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:33:28.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:33:28.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpw30k0laj:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpair9xpj8:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:33:28.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: cephadm 2024-01-15T22:33:27.480596+0000 mgr.smithi099.cmjeyn (mgr.14184) 230 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-15T22:33:28.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: service_id: all-available-devices 2024-01-15T22:33:28.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: service_name: osd.all-available-devices 2024-01-15T22:33:28.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: placement: 2024-01-15T22:33:28.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: host_pattern: '*' 2024-01-15T22:33:28.407 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: spec: 2024-01-15T22:33:28.407 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: data_devices: 2024-01-15T22:33:28.407 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: all: true 2024-01-15T22:33:28.407 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: filter_logic: AND 2024-01-15T22:33:28.407 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: objectstore: bluestore 2024-01-15T22:33:28.407 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:33:28.407 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpf44ann54:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpsdpvzxui:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:33:28.407 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:33:28.407 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:33:28.407 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:33:28.407 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 683f5818-c5ab-40af-8d03-070567658d0e 2024-01-15T22:33:28.407 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:33:28.407 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:33:28.408 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:33:28.408 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:33:28.408 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:33:28.408 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr self.prepare() 2024-01-15T22:33:28.408 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:33:28.408 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:33:28.408 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:33:28.408 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:33:28.408 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:33:28.408 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:33:28.408 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:33:28.408 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:33:28.408 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:33:28.408 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:33:28.409 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:33:28.409 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:33:28.409 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:33:28.409 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:33:28.409 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:33:28.409 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:33:28.409 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:33:28.409 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:33:28.409 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:33:28.409 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:33:28.409 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:33:28.410 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:33:28.410 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:33:28.410 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: std 2024-01-15T22:33:28.410 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: err File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:33:28.410 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:33:28.410 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:33:28.410 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:33:28.410 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:33:28.411 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:33:28.411 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:33:28.411 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:33:28.411 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:33:28.411 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:33:28.411 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:33:28.411 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:33:28.411 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:33:28.411 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:33:28.411 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:33:28.411 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:33:28.411 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:33:28.411 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:33:28.412 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:33:28.412 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:33:28.412 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:33:28.412 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:33:28.412 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:33:28.412 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:33:28.412 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:33:28.412 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:33:28.412 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:33:28.413 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:33:28.413 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:33:28.413 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: main() 2024-01-15T22:33:28.413 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:33:28.413 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: r = ctx.func(ctx) 2024-01-15T22:33:28.413 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:33:28.413 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:33:28.413 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:33:28.413 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:33:28.413 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:33:28.413 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:33:28.413 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:33:28.414 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:33:28.414 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:33:28.414 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:33:28.414 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:33:28.414 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:33:28.414 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpf44ann54:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpsdpvzxui:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:33:28.414 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:33:28.414 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-15T22:33:28.414 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: if self._apply_service(spec): 2024-01-15T22:33:28.414 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-15T22:33:28.414 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-15T22:33:28.414 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-15T22:33:28.415 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-15T22:33:28.415 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-15T22:33:28.415 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-15T22:33:28.415 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-15T22:33:28.415 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-15T22:33:28.415 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-15T22:33:28.415 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: raise self._value 2024-01-15T22:33:28.415 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-15T22:33:28.415 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: result = (True, func(*args, **kwds)) 2024-01-15T22:33:28.415 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-15T22:33:28.415 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: return list(map(*args)) 2024-01-15T22:33:28.415 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:33:28.415 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: return f(*arg) 2024-01-15T22:33:28.415 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:33:28.416 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:33:28.416 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:33:28.416 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: code, '\n'.join(err))) 2024-01-15T22:33:28.416 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:33:28.416 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpf44ann54:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpsdpvzxui:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:33:28.416 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:33:28.416 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:33:28.416 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:33:28.416 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd ne 2024-01-15T22:33:28.416 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: w 683f5818-c5ab-40af-8d03-070567658d0e 2024-01-15T22:33:28.416 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:33:28.416 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:33:28.416 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:33:28.417 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:33:28.417 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:33:28.417 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr self.prepare() 2024-01-15T22:33:28.417 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:33:28.417 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:33:28.417 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:33:28.417 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:33:28.417 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:33:28.417 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:33:28.417 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:33:28.417 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:33:28.417 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:33:28.417 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:33:28.417 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:33:28.417 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:33:28.417 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:33:28.418 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:33:28.418 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:33:28.418 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:33:28.418 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:33:28.418 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:33:28.418 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:33:28.418 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:33:28.418 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:33:28.418 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:33:28.418 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:33:28.418 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:33:28.418 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:33:28.418 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:33:28.418 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:33:28.418 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:33:28.419 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:33:28.419 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:33:28.419 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:33:28.419 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:33:28.419 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:33:28.419 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:33:28.419 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:33:28.419 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:33:28.419 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:33:28.419 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:33:28.419 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:33:28.419 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:33:28.419 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:33:28.419 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:33:28.419 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:33:28.420 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:33:28.420 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:33:28.420 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:33:28.420 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:33:28.420 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:33:28.420 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:33:28.420 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:33:28.420 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:33:28.420 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:33:28.420 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: main() 2024-01-15T22:33:28.420 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:33:28.420 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: r = ctx.func(ctx) 2024-01-15T22:33:28.420 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:33:28.420 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:33:28.420 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:33:28.421 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:33:28.421 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:33:28.421 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:33:28.421 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:33:28.421 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:33:28.421 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:33:28.421 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:33:28.421 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:33:28.421 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:33:28.421 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpf44ann54:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpsdpvzxui:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:33:28.421 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:28 smithi099 conmon[35292]: cluster 2024-01-15T22:33:27.481094+0000 mgr.smithi099.cmjeyn (mgr.14184) 231 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:28.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: cephadm 2024-01-15T22:33:27.004663+0000 mgr.smithi099.cmjeyn (mgr.14184) 227 : cephadm [ERR] 2024-01-15T22:33:28.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi162', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-15T22:33:28.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:33:28.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:33:28.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: return f(*arg) 2024-01-15T22:33:28.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:33:28.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:33:28.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:33:28.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: code, '\n'.join(err))) 2024-01-15T22:33:28.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:33:28.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpf44ann54:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpsdpvzxui:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:33:28.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:33:28.449 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:33:28.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:33:28.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 683f5818-c5ab-40af-8d03-070567658d0e 2024-01-15T22:33:28.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:33:28.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:33:28.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:33:28.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:33:28.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:33:28.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr self.prepare() 2024-01-15T22:33:28.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:33:28.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:33:28.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:33:28.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:33:28.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:33:28.450 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:33:28.451 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:33:28.451 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:33:28.451 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:33:28.451 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:33:28.451 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:33:28.451 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:33:28.451 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:33:28.451 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:33:28.451 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:33:28.451 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:33:28.451 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:33:28.451 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:33:28.451 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:33:28.452 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:33:28.452 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:33:28.452 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:33:28.452 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:33:28.452 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:33:28.452 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:33:28.452 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:33:28.452 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:33:28.452 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:33:28.452 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:33:28.452 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:33:28.452 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:33:28.452 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:33:28.452 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:33:28.453 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:33:28.453 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:33:28.453 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:33:28.453 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:33:28.453 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:33:28.453 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:33:28.453 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:33:28.453 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:33:28.453 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:33:28.453 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:33:28.453 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:33:28.453 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:33:28.453 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:33:28.453 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:33:28.453 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:33:28.453 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:33:28.454 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:33:28.454 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:33:28.454 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:33:28.454 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: main() 2024-01-15T22:33:28.454 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:33:28.454 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: r = ctx.func(ctx) 2024-01-15T22:33:28.454 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:33:28.454 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:33:28.454 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:33:28.454 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:33:28.454 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:33:28.454 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:33:28.454 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 192 2024-01-15T22:33:28.454 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: 7, in _validate_fsid 2024-01-15T22:33:28.455 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:33:28.455 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:33:28.455 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:33:28.455 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:33:28.455 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:33:28.455 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpf44ann54:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpsdpvzxui:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:33:28.455 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: cluster 2024-01-15T22:33:27.374312+0000 mgr.smithi099.cmjeyn (mgr.14184) 228 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:28.455 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: cephadm 2024-01-15T22:33:27.479500+0000 mgr.smithi099.cmjeyn (mgr.14184) 229 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi162', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-15T22:33:28.455 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:33:28.455 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:33:28.455 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: return f(*arg) 2024-01-15T22:33:28.455 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:33:28.455 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:33:28.456 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:33:28.456 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: code, '\n'.join(err))) 2024-01-15T22:33:28.456 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi099/config 2024-01-15T22:33:28.456 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpw30k0laj:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpair9xpj8:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:33:28.456 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:33:28.456 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:33:28.456 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:33:28.456 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new e7735357-e3f2-4fd3-acab-4220dcdebd43 2024-01-15T22:33:28.456 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:33:28.456 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:33:28.456 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:33:28.456 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:33:28.456 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:33:28.456 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr self.prepare() 2024-01-15T22:33:28.457 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:33:28.457 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:33:28.457 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:33:28.457 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:33:28.457 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:33:28.457 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:33:28.457 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:33:28.457 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:33:28.457 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:33:28.457 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:33:28.457 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:33:28.457 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:33:28.457 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:33:28.457 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:33:28.457 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:33:28.458 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:33:28.458 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:33:28.458 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:33:28.458 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:33:28.458 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:33:28.458 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:33:28.459 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:33:28.459 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:33:28.459 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:33:28.459 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:33:28.460 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:33:28.460 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:33:28.460 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:33:28.460 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:33:28.460 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:33:28.460 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:33:28.460 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:33:28.460 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:33:28.460 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:33:28.460 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:33:28.460 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:33:28.460 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:33:28.461 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:33:28.461 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:33:28.461 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packag 2024-01-15T22:33:28.461 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: es/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:33:28.461 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:33:28.461 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:33:28.461 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:33:28.461 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:33:28.461 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:33:28.461 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:33:28.461 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:33:28.461 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:33:28.461 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:33:28.461 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:33:28.462 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:33:28.462 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:33:28.462 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: main() 2024-01-15T22:33:28.462 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:33:28.462 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: r = ctx.func(ctx) 2024-01-15T22:33:28.462 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:33:28.462 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:33:28.462 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:33:28.462 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:33:28.462 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:33:28.462 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:33:28.462 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:33:28.462 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:33:28.462 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:33:28.463 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:33:28.463 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:33:28.463 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:33:28.463 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpw30k0laj:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpair9xpj8:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:33:28.463 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: cephadm 2024-01-15T22:33:27.480596+0000 mgr.smithi099.cmjeyn (mgr.14184) 230 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-15T22:33:28.463 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: service_id: all-available-devices 2024-01-15T22:33:28.463 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: service_name: osd.all-available-devices 2024-01-15T22:33:28.463 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: placement: 2024-01-15T22:33:28.463 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: host_pattern: '*' 2024-01-15T22:33:28.463 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: spec: 2024-01-15T22:33:28.463 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: data_devices: 2024-01-15T22:33:28.463 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: all: true 2024-01-15T22:33:28.463 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: filter_logic: AND 2024-01-15T22:33:28.463 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: objectstore: bluestore 2024-01-15T22:33:28.463 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:33:28.463 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpf44ann54:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpsdpvzxui:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:33:28.464 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:33:28.464 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:33:28.464 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:33:28.464 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 683f5818-c5ab-40af-8d03-070567658d0e 2024-01-15T22:33:28.464 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:33:28.464 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:33:28.464 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:33:28.464 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:33:28.464 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:33:28.464 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr self.prepare() 2024-01-15T22:33:28.464 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:33:28.464 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:33:28.464 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:33:28.464 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:33:28.464 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:33:28.465 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:33:28.465 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:33:28.465 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:33:28.465 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:33:28.465 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:33:28.465 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:33:28.465 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:33:28.465 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:33:28.465 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:33:28.465 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:33:28.465 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:33:28.465 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:33:28.465 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:33:28.465 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:33:28.465 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:33:28.465 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:33:28.466 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:33:28.466 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:33:28.466 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/ 2024-01-15T22:33:28.466 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:33:28.466 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:33:28.466 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:33:28.466 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:33:28.466 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:33:28.466 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:33:28.466 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:33:28.466 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:33:28.466 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:33:28.466 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:33:28.466 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:33:28.466 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:33:28.466 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:33:28.467 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:33:28.467 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:33:28.467 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:33:28.467 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:33:28.467 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:33:28.467 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:33:28.467 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:33:28.467 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:33:28.467 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:33:28.467 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:33:28.467 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:33:28.467 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:33:28.467 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:33:28.467 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:33:28.467 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:33:28.467 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:33:28.468 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: main() 2024-01-15T22:33:28.468 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:33:28.468 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: r = ctx.func(ctx) 2024-01-15T22:33:28.468 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:33:28.468 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:33:28.468 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:33:28.468 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:33:28.468 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:33:28.468 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:33:28.468 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:33:28.468 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:33:28.468 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:33:28.468 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:33:28.468 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:33:28.468 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:33:28.468 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpf44ann54:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpsdpvzxui:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:33:28.469 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:33:28.469 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-15T22:33:28.469 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: if self._apply_service(spec): 2024-01-15T22:33:28.469 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-15T22:33:28.469 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-15T22:33:28.469 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-15T22:33:28.469 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-15T22:33:28.469 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-15T22:33:28.469 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-15T22:33:28.469 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-15T22:33:28.469 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-15T22:33:28.469 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-15T22:33:28.469 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: raise self._value 2024-01-15T22:33:28.469 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-15T22:33:28.469 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: result = (True, func(*args, **kwds)) 2024-01-15T22:33:28.469 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-15T22:33:28.470 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: return list(map(*args)) 2024-01-15T22:33:28.470 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:33:28.470 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: return f(*arg) 2024-01-15T22:33:28.470 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:33:28.470 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:33:28.470 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:33:28.470 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: code, '\n'.join(err))) 2024-01-15T22:33:28.470 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:33:28.470 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpf44ann54:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpsdpvzxui:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:33:28.470 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:33:28.470 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:33:28.470 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:33:28.470 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 683f5818-c5 2024-01-15T22:33:28.470 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: ab-40af-8d03-070567658d0e 2024-01-15T22:33:28.471 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:33:28.471 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:33:28.471 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:33:28.471 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:33:28.471 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:33:28.471 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr self.prepare() 2024-01-15T22:33:28.471 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:33:28.471 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:33:28.471 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:33:28.471 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:33:28.471 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:33:28.471 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:33:28.471 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:33:28.471 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:33:28.471 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:33:28.472 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:33:28.472 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:33:28.472 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:33:28.472 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:33:28.472 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:33:28.472 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:33:28.472 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:33:28.472 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:33:28.472 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:33:28.472 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:33:28.472 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:33:28.473 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:33:28.473 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:33:28.473 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:33:28.473 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:33:28.473 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:33:28.473 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:33:28.473 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:33:28.473 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:33:28.473 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:33:28.473 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:33:28.473 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:33:28.473 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:33:28.473 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:33:28.473 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:33:28.473 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:33:28.474 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:33:28.474 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:33:28.474 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:33:28.474 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:33:28.474 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:33:28.474 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:33:28.474 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:33:28.474 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:33:28.474 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:33:28.474 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:33:28.474 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:33:28.474 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:33:28.474 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:33:28.475 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:33:28.475 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:33:28.475 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:33:28.475 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:33:28.475 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: main() 2024-01-15T22:33:28.475 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:33:28.475 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: r = ctx.func(ctx) 2024-01-15T22:33:28.475 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:33:28.475 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:33:28.475 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:33:28.475 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:33:28.475 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:33:28.475 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:33:28.475 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:33:28.475 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:33:28.476 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:33:28.476 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:33:28.476 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:33:28.476 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:33:28.476 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpf44ann54:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpsdpvzxui:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:33:28.476 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:28 smithi162 conmon[39755]: cluster 2024-01-15T22:33:27.481094+0000 mgr.smithi099.cmjeyn (mgr.14184) 231 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:29.347 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:29 smithi099 conmon[35292]: cluster 2024-01-15T22:33:28.149391+0000 mon.smithi099 (mon.0) 598 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-15T22:33:29.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:29 smithi162 conmon[39755]: cluster 2024-01-15T22:33:28.149391+0000 mon.smithi099 (mon.0) 598 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-15T22:33:29.667 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:33:30.236 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:33:30.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:30 smithi162 conmon[39755]: cluster 2024-01-15T22:33:29.481469+0000 mgr.smithi099.cmjeyn (mgr.14184) 232 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:30.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:30 smithi162 conmon[39755]: 2024-01-15T22:33:30.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:30 smithi162 conmon[39755]: audit 2024-01-15T22:33:29.666844+0000 mon.smithi099 (mon.0) 599 : audit [DBG] from='client.? 172.21.15.99:0/2861938568' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:33:30.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:30 smithi099 conmon[35292]: cluster 2024-01-15T22:33:29.481469+0000 mgr.smithi099.cmjeyn (mgr.14184) 232 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:30.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:30 smithi099 conmon[35292]: audit 2024-01- 2024-01-15T22:33:30.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:30 smithi099 conmon[35292]: 15T22:33:29.666844+0000 mon.smithi099 (mon.0) 599 : audit [DBG] from='client.? 172.21.15.99:0/2861938568' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:33:31.236 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:33:32.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:32 smithi099 conmon[35292]: cluster 2024-01-15T22:33:31.481984+0000 mgr.smithi099.cmjeyn (mgr.14184) 233 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:32.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:32 smithi162 conmon[39755]: cluster 2024-01-15T22:33:31.481984+0000 mgr.smithi099.cmjeyn (mgr.14184) 233 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:33.084 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:33:33.687 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:33:33.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:33 smithi099 conmon[35292]: audit 2024-01-15T22:33:33.083774+0000 mon.smithi099 (mon.0) 600 : audit [DBG] 2024-01-15T22:33:33.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:33 smithi099 conmon[35292]: from='client.? 172.21.15.99:0/2162611508' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:33:33.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:33 smithi162 conmon[39755]: audit 2024-01-15T22:33:33.083774+0000 mon.smithi099 (mon.0) 600 : audit [DBG] from='client.? 172.21.15.99:0/2162611508' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:33:34.688 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:33:34.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:34 smithi099 conmon[35292]: cluster 2024-01-15T22:33:33.482389+0000 mgr.smithi099.cmjeyn (mgr.14184) 234 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:34.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:34 smithi162 conmon[39755]: cluster 2024-01-15T22:33:33.482389+0000 mgr.smithi099.cmjeyn (mgr.14184) 234 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:36.175 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:35 smithi099 conmon[35292]: cluster 2024-01-15T22:33:35.482996+0000 mgr.smithi099.cmjeyn (mgr.14184) 235 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:36.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:35 smithi162 conmon[39755]: cluster 2024-01-15T22:33:35.482996+0000 mgr.smithi099.cmjeyn (mgr.14184) 235 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:36.493 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:33:37.036 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:33:37.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:36 smithi099 conmon[35292]: audit 2024-01-15T22:33:36.492959+0000 mon.smithi099 (mon.0) 601 : audit [DBG] from='client.? 172.21.15.99:0/284585393' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:33:37.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:36 smithi162 conmon[39755]: audit 2024-01-15T22:33:36.492959+0000 mon.smithi099 (mon.0) 601 : audit [DBG] from='client.? 172.21.15.99:0/284585393' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:33:38.038 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:33:38.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:37 smithi099 conmon[35292]: cluster 2024-01-15T22:33:37.483401+0000 mgr.smithi099.cmjeyn (mgr.14184) 236 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:38.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:37 smithi162 conmon[39755]: cluster 2024-01-15T22:33:37.483401+0000 mgr.smithi099.cmjeyn (mgr.14184) 236 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:39.773 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:33:40.327 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:33:40.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:40 smithi099 conmon[35292]: cluster 2024-01-15T22:33:39.483868+0000 mgr.smithi099.cmjeyn (mgr.14184) 237 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:40.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:40 smithi099 conmon[35292]: audit 2024-01-15T22:33:39.772349+0000 mon.smithi099 (mon.0) 602 : audit [DBG] from='client.? 172.21.15.99:0/2337778136' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:33:40.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:40 smithi162 conmon[39755]: cluster 2024-01-15T22:33:39.483868+0000 mgr.smithi099.cmjeyn (mgr.14184) 237 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:40.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:40 smithi162 conmon[39755]: audit 2024-01-15T22:33:39.772349+0000 mon.smithi099 (mon.0) 602 : audit [DBG] from='client.? 172.21.15.99:0/2337778136' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:33:41.328 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:33:42.795 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:42 smithi099 conmon[35292]: cluster 2024-01-15T22:33:41.484269+0000 mgr.smithi099.cmjeyn (mgr.14184 2024-01-15T22:33:42.795 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:42 smithi099 conmon[35292]: ) 238 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:42.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:42 smithi162 conmon[39755]: cluster 2024-01-15T22:33:41.484269+0000 mgr.smithi099.cmjeyn (mgr.14184) 238 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:43.133 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:33:43.693 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:33:43.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:43 smithi099 conmon[35292]: audit 2024-01-15T22:33:43.133349+0000 mon.smithi099 (mon.0) 603 : audit [DBG] from='client.? 172.21.15.99:0/3071311152' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:33:43.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:43 smithi162 conmon[39755]: audit 2024-01-15T22:33:43.133349+0000 mon.smithi099 (mon.0) 603 : audit [DBG] from='client.? 172.21.15.99:0/3071311152' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:33:44.694 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:33:44.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:44 smithi099 conmon[35292]: cluster 2024-01-15T22:33:43.484941+0000 mgr.smithi099.cmjeyn (mgr.14184) 239 : cluster [DBG] 2024-01-15T22:33:44.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:44 smithi099 conmon[35292]: pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:44.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:44 smithi162 conmon[39755]: cluster 2024-01-15T22:33:43.484941+0000 mgr.smithi099.cmjeyn (mgr.14184) 239 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:46.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:45 smithi099 conmon[35292]: cluster 2024-01-15T22:33:45.485613+0000 mgr.smithi099.cmjeyn (mgr.14184) 2024-01-15T22:33:46.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:45 smithi099 conmon[35292]: 240 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:46.408 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:33:46.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:45 smithi162 conmon[39755]: cluster 2024-01-15T22:33:45.485613+0000 mgr.smithi099.cmjeyn (mgr.14184) 240 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:46.910 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:33:47.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:46 smithi099 conmon[35292]: audit 2024-01-15T22:33:46.407733+0000 mon.smithi099 (mon.0) 604 : audit [DBG] from='client.? 172.21.15.99:0/743884301' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:33:47.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:46 smithi162 conmon[39755]: audit 2024-01-15T22:33:46.407733+0000 mon.smithi099 (mon.0) 604 : audit [DBG] from='client.? 172.21.15.99:0/743884301' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:33:47.912 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:33:48.352 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:48 smithi099 conmon[35292]: cluster 2024-01-15T22:33:47.486020+0000 mgr.smithi099.cmjeyn (mgr.14184) 241 : 2024-01-15T22:33:48.352 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:48 smithi099 conmon[35292]: cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:48.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:48 smithi162 conmon[39755]: cluster 2024-01-15T22:33:47.486020+0000 mgr.smithi099.cmjeyn (mgr.14184) 241 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:49.691 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:33:50.208 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:33:50.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:50 smithi099 conmon[35292]: cluster 2024-01-15T22:33:49.486383+0000 mgr.smithi099.cmjeyn (mgr.14184) 242 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:50.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:50 smithi099 conmon[35292]: audit 2024-01-15T22:33:49.690082+0000 mon.smithi099 (mon.0) 605 : audit [DBG] from='client.? 172.21.15.99:0/1460882146' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:33:50.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:50 smithi162 conmon[39755]: cluster 2024-01-15T22:33:49.486383+0000 mgr.smithi099.cmjeyn (mgr.14184) 242 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:50.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:50 smithi162 conmon[39755]: audit 2024-01-15T22:33:49.690082+0000 mon.smithi099 (mon.0) 605 : audit [DBG] from='client.? 172.21.15.99:0/1460882146' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:33:51.209 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:33:52.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:52 smithi099 conmon[35292]: cluster 2024-01-15T22:33:51.486750+0000 mgr.smithi099.cmjeyn (mgr.14184) 243 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:52.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:52 smithi162 conmon[39755]: cluster 2024-01-15T22:33:51.486750+0000 mgr.smithi099.cmjeyn (mgr.14184) 243 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:52.979 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:33:53.533 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:33:53.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:53 smithi099 conmon[35292]: audit 2024-01-15T22:33:52.978197+0000 mon.smithi099 (mon.0) 606 : audit [DBG] from='client.? 172.21.15.99:0/2254542128' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:33:53.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:53 smithi162 conmon[39755]: audit 2024-01-15T22:33:52.978197+0000 mon.smithi099 (mon.0) 606 : audit [DBG] from='client.? 172.21.15.99:0/2254542128' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:33:54.534 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:33:54.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:54 smithi099 conmon[35292]: cluster 2024-01-15T22:33:53.487134+0000 mgr.smithi099.cmjeyn (mgr.14184) 244 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:54.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:54 smithi162 conmon[39755]: cluster 2024-01-15T22:33:53.487134+0000 mgr.smithi099.cmjeyn (mgr.14184) 244 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:56.224 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:33:56.249 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:55 smithi099 conmon[35292]: cluster 2024-01-15T22:33:55.487845+0000 mgr.smithi099.cmjeyn (mgr 2024-01-15T22:33:56.249 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:55 smithi099 conmon[35292]: .14184) 245 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:56.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:55 smithi162 conmon[39755]: cluster 2024-01-15T22:33:55.487845+0000 mgr.smithi099.cmjeyn (mgr.14184) 245 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:56.752 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:33:57.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:56 smithi099 conmon[35292]: audit 2024-01-15T22:33:56.223097+0000 mon.smithi099 (mon.0) 607 : audit [DBG] from='client.? 172.21.15.99:0/1762511701' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:33:57.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:56 smithi162 conmon[39755]: audit 2024-01-15T22:33:56.223097+0000 mon.smithi099 (mon.0) 607 : audit [DBG] from='client.? 172.21.15.99:0/1762511701' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:33:57.754 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:33:58.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:58 smithi099 conmon[35292]: cluster 2024-01-15T22:33:57.488522+0000 mgr.smithi099.cmjeyn (mgr.14184) 246 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:58.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:58 smithi162 conmon[39755]: cluster 2024-01-15T22:33:57.488522+0000 mgr.smithi099.cmjeyn (mgr.14184) 246 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:33:59.469 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:33:59.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:33:59 smithi099 conmon[35292]: audit 2024-01-15T22:33:59.468859+0000 mon.smithi099 (mon.0) 608 : audit [DBG] from='client.? 172.21.15.99:0/3797347563' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:33:59.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:33:59 smithi162 conmon[39755]: audit 2024-01-15T22:33:59.468859+0000 mon.smithi099 (mon.0) 608 : audit [DBG] from='client.? 172.21.15.99:0/3797347563' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:34:00.049 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:34:00.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:00 smithi099 conmon[35292]: cluster 2024-01-15T22:33:59.488981+0000 mgr.smithi099.cmjeyn (mgr.14184) 247 : cluster [DBG] 2024-01-15T22:34:00.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:00 smithi099 conmon[35292]: pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:00.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:00 smithi162 conmon[39755]: cluster 2024-01-15T22:33:59.488981+0000 mgr.smithi099.cmjeyn (mgr.14184) 247 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:01.050 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:34:02.804 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:34:02.831 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:02 smithi099 conmon[35292]: cluster 2024-01-15T22:34:01.489595+0000 mgr.smithi099.cmjeyn (mgr.14184) 248 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:02.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:02 smithi162 conmon[39755]: cluster 2024-01-15T22:34:01.489595+0000 mgr.smithi099.cmjeyn (mgr.14184) 248 : cluster [DBG] 2024-01-15T22:34:02.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:02 smithi162 conmon[39755]: pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:03.331 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:34:03.654 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:03 smithi099 conmon[35292]: audit 2024-01-15T22:34:02.803492+0000 mon.smithi099 (mon.0) 609 : audit [DBG] from='client.? 172.21.15.99:0/1139556579' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:34:03.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:03 smithi162 conmon[39755]: audit 2024-01-15T22:34:02.803492+0000 mon.smithi099 (mon.0) 609 : audit [DBG] from='client.? 172.21.15.99:0/1139556579' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:34:04.332 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:34:04.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:04 smithi099 conmon[35292]: cluster 2024-01-15T22:34:03.490047+0000 mgr.smithi099.cmjeyn (mgr.14184) 249 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:04.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:04 smithi162 conmon[39755]: cluster 2024-01-15T22:34:03.490047+0000 mgr.smithi099.cmjeyn (mgr.14184) 249 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:06.034 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:34:06.248 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:05 smithi099 conmon[35292]: cluster 2024-01-15T22:34:05.490451+0000 mgr.smithi099.cmjeyn (mgr.14184) 250 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:06.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:05 smithi162 conmon[39755]: cluster 2024-01-15T22:34:05.490451+0000 mgr.smithi099.cmjeyn (mgr.14184) 250 : cluster 2024-01-15T22:34:06.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:05 smithi162 conmon[39755]: [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:06.609 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:34:07.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:06 smithi099 conmon[35292]: audit 2024-01-15T22:34:06.033430+0000 mon.smithi099 (mon.0) 610 : audit [DBG] from='client.? 172.21.15.99:0/3604087902' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:34:07.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:07 smithi162 conmon[39755]: audit 2024-01-15T22:34:06.033430+0000 mon.smithi099 (mon.0) 610 : audit [DBG] from='client.? 172.21.15.99:0/3604087902' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:34:07.610 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:34:08.404 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:08 smithi099 conmon[35292]: cluster 2024-01-15T22:34:07.491120+0000 mgr.smithi099.cmjeyn (mgr.14184) 251 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:08.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:08 smithi162 conmon[39755]: cluster 2024-01-15T22:34:07.491120+0000 mgr.smithi099.cmjeyn (mgr.14184) 251 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:09.345 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:34:09.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:09 smithi099 conmon[35292]: audit 2024-01-15T22:34:09.344042+0000 mon.smithi099 (mon.0) 611 : audit [DBG] from='client.? 172.21.15.99:0/1677396318' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:34:09.694 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:09 smithi162 conmon[39755]: audit 2024-01-15T22:34:09.344042+0000 mon.smithi099 (mon.0) 611 : audit [DBG] from='client.? 172.21.15.99:0/1677396318' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:34:09.863 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:34:10.697 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:10 smithi162 conmon[39755]: cluster 2024-01-15T22:34:09.491609+0000 mgr.smithi099.cmjeyn (mgr.14184) 252 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:10.864 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:34:10.897 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:10 smithi099 conmon[35292]: cluster 2024-01-15T22:34:09.491609+0000 mgr.smithi099.cmjeyn (mgr.14184) 252 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:12.596 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:34:12.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:12 smithi099 conmon[35292]: cluster 2024-01-15T22: 2024-01-15T22:34:12.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:12 smithi099 conmon[35292]: 34:11.492072+0000 mgr.smithi099.cmjeyn (mgr.14184) 253 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:12.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:12 smithi162 conmon[39755]: cluster 2024-01-15T22:34:11.492072+0000 mgr.smithi099.cmjeyn (mgr.14184) 253 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:13.163 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:34:13.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:13 smithi099 conmon[35292]: audit 2024-01-15T22:34:12.595508+0000 mon.smithi099 (mon.0) 612 : audit [DBG] from='client.? 172.21.15.99:0/3782830552' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:34:13.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:13 smithi162 conmon[39755]: audit 2024-01-15T22:34:12.595508+0000 mon.smithi099 (mon.0) 612 : audit [DBG] from='client.? 172.21.15.99:0/3782830552' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:34:14.164 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:34:14.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:14 smithi099 conmon[35292]: cluster 2024-01-15T22:34:13. 2024-01-15T22:34:14.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:14 smithi099 conmon[35292]: 492841+0000 mgr.smithi099.cmjeyn (mgr.14184) 254 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:14.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:14 smithi162 conmon[39755]: cluster 2024-01-15T22:34:13.492841+0000 mgr.smithi099.cmjeyn (mgr.14184) 254 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:15.927 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:34:16.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:15 smithi099 conmon[35292]: cluster 2024-01-15T22:34:15.493183+0000 mgr.smithi099.cmjeyn (mgr.14184) 255 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:16.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:15 smithi099 conmon[35292]: audit 2024-01-15T22:34:15.926663+0000 mon.smithi099 (mon.0) 613 : audit [DBG] from='client.? 172.21.15.99:0/2453170439' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:34:16.429 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:34:16.447 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:15 smithi162 conmon[39755]: cluster 2024-01-15T22:34:15.493183+0000 mgr.smithi099.cmjeyn (mgr.14184) 255 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:16.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:15 smithi162 conmon[39755]: audit 2024-01-15T22:34:16.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:15 smithi162 conmon[39755]: 2024-01-15T22:34:15.926663+0000 mon.smithi099 (mon.0) 613 : audit [DBG] from='client.? 172.21.15.99:0/2453170439' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:34:17.430 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:34:18.772 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:18 smithi099 conmon[35292]: cluster 2024-01-15T22:34:17.493621+0000 mgr.smithi099.cmjeyn (mgr.14184) 256 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:18.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:18 smithi162 conmon[39755]: cluster 2024-01-15T22:34:17.493621+0000 mgr.smithi099.cmjeyn (mgr.14184) 256 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:19.156 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:34:19.737 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:34:19.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:19 smithi099 conmon[35292]: audit 2024-01-15T22:34:19.156005+0000 mon.smithi099 (mon.0) 614 : audit [DBG] from='client.? 172.21.15.99:0/1590593021' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:34:19.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:19 smithi162 conmon[39755]: audit 2024-01-15T22:34:19.156005+0000 mon.smithi099 (mon.0) 614 : audit [DBG] from='client.? 172.21.15.99:0/1590593021' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:34:20.738 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:34:20.904 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:20 smithi099 conmon[35292]: cluster 2024-01-15T22:34:19.494062+0000 mgr.smithi099.cmjeyn (mgr.14184) 257 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:20.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:20 smithi162 conmon[39755]: cluster 2024-01-15T22:34:19.494062+0000 mgr.smithi099.cmjeyn (mgr.14184) 257 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:22.529 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:34:22.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:22 smithi099 conmon[35292]: cluster 2024-01-15T22:34:21.494502+0000 mgr.smithi099.cmjeyn (mgr.14184 2024-01-15T22:34:22.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:22 smithi099 conmon[35292]: ) 258 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:22.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:22 smithi099 conmon[35292]: audit 2024-01-15T22:34:22.528694+0000 mon.smithi099 (mon.0) 615 : audit [DBG] from='client.? 172.21.15.99:0/2606471436' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:34:22.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:22 smithi162 conmon[39755]: cluster 2024-01-15T22:34:21.494502+0000 mgr.smithi099.cmjeyn (mgr.14184) 258 2024-01-15T22:34:22.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:22 smithi162 conmon[39755]: : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:22.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:22 smithi162 conmon[39755]: audit 2024-01-15T22:34:22.528694+0000 mon.smithi099 (mon.0) 615 : audit [DBG] from='client.? 172.21.15.99:0/2606471436' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:34:23.121 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:34:24.122 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:34:24.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:24 smithi099 conmon[35292]: cluster 2024-01-15T22:34:23.495136+0000 mgr.smithi099.cmjeyn (mgr.14184) 2024-01-15T22:34:24.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:24 smithi099 conmon[35292]: 259 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:24.947 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:24 smithi162 conmon[39755]: cluster 2024-01-15T22:34:23.495136+0000 mgr.smithi099.cmjeyn (mgr.14184) 259 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:25.908 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:34:26.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:26 smithi099 conmon[35292]: cluster 2024-01-15T22:34:25.495530+0000 mgr.smithi099.cmjeyn (mgr.14184) 260 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:26.405 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:26 smithi099 conmon[35292]: audit 2024-01-15T22:34:25.907485+0000 mon.smithi099 (mon.0) 616 : audit [DBG] from='client.? 172.21.15.99:0/801292805' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:34:26.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:26 smithi162 conmon[39755]: cluster 2024-01-15T22:34:25.495530+0000 mgr.smithi099.cmjeyn (mgr.14184) 260 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:26.448 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:26 smithi162 conmon[39755]: audit 2024-01-15T22:34:25.907485+0000 mon.smithi099 (mon.0) 616 : audit [DBG] from='client.? 172.21.15.99:0/801292805' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:34:26.476 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:34:27.477 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:34:27.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:27 smithi099 conmon[35292]: audit 2024-01-15T22:34:27.482359+0000 mon.smithi099 (mon. 2024-01-15T22:34:27.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:27 smithi099 conmon[35292]: 0) 617 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-15T22:34:27.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:27 smithi162 conmon[39755]: audit 2024-01-15T22:34:27.482359+0000 mon.smithi099 (mon.0) 617 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-15T22:34:28.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:28 smithi099 conmon[35292]: cluster 2024-01-15T22:34:27.495993+0000 mgr.smithi099.cmjeyn (mgr.14184) 261 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:28.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:28 smithi099 conmon[35292]: audit 2024-01-15T22:34:27.840851+0000 mon.smithi099 (mon.0) 618 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:34:28.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:28 smithi099 conmon[35292]: audit 2024-01-15T22:34:28.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:28 smithi099 conmon[35292]: 2024-01-15T22:34:27.841447+0000 mon.smithi099 (mon.0 2024-01-15T22:34:28.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:28 smithi099 conmon[35292]: ) 619 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-15T22:34:28.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:28 smithi099 conmon[35292]: audit 2024-01- 2024-01-15T22:34:28.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:28 smithi099 conmon[35292]: 15T22:34:27.847712+0000 2024-01-15T22:34:28.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:28 smithi099 conmon[35292]: mon.smithi099 (mon.0) 620 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:34:28.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:28 smithi099 conmon[35292]: audit 2024-01-15T22:34:28.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:28 smithi099 conmon[35292]: 2024-01-15T22: 2024-01-15T22:34:28.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:28 smithi099 conmon[35292]: 34:27.851492+0000 mon.smithi099 (mon.0) 2024-01-15T22:34:28.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:28 smithi099 conmon[35292]: 621 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-15T22:34:28.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:28 smithi099 conmon[35292]: audit 2024-01-15T22:34:28.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:28 smithi099 conmon[35292]: 2024-01-15T22 2024-01-15T22:34:28.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:28 smithi099 conmon[35292]: :34:27.854115+0000 mon.smithi099 (mon.0 2024-01-15T22:34:28.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:28 smithi099 conmon[35292]: ) 622 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-15T22:34:28.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:28 smithi099 conmon[35292]: audit 2024-01-15T22:34:28.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:28 smithi099 conmon[35292]: 2024-01-15T 2024-01-15T22:34:28.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:28 smithi099 conmon[35292]: 22:34:27.854974+0000 mon.smithi099 (mon.0 2024-01-15T22:34:28.657 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:28 smithi099 conmon[35292]: ) 623 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-15T22:34:28.657 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:28 smithi099 conmon[35292]: audit 2024-01-15T22:34:28.657 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:28 smithi099 conmon[35292]: 2024-01-15T22 2024-01-15T22:34:28.657 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:28 smithi099 conmon[35292]: :34:27.855141+0000 mon.smithi099 (mon.0 2024-01-15T22:34:28.657 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:28 smithi099 conmon[35292]: ) 624 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:34:28.657 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:28 smithi099 conmon[35292]: audit 2024-01 2024-01-15T22:34:28.657 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:28 smithi099 conmon[35292]: -15T22:34:27.855570 2024-01-15T22:34:28.657 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:28 smithi099 conmon[35292]: +0000 mon.smithi099 (mon.0) 625 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:34:28.657 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:28 smithi099 conmon[35292]: 2024-01-15T22:34:28.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:28 smithi162 conmon[39755]: cluster 2024-01-15T22:34:27.495993+0000 mgr.smithi099.cmjeyn (mgr.14184) 261 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:28.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:28 smithi162 conmon[39755]: audit 2024-01-15T22:34:27.840851+0000 mon.smithi099 (mon.0) 618 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:34:28.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:28 smithi162 conmon[39755]: audit 2024-01-15T22:34:27.841447+0000 mon.smithi099 (mon.0) 619 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-15T22:34:28.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:28 smithi162 conmon[39755]: audit 2024-01-15T22:34:27.847712+0000 mon.smithi099 (mon.0) 620 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:34:28.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:28 smithi162 conmon[39755]: audit 2024-01-15T22:34:27.851492+0000 mon.smithi099 (mon.0) 621 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-15T22:34:28.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:28 smithi162 conmon[39755]: audit 2024-01-15T22:34:27.854115+0000 mon.smithi099 (mon.0) 622 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-15T22:34:28.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:28 smithi162 conmon[39755]: audit 2024-01-15T22:34:27.854974+0000 mon.smithi099 (mon.0) 623 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-15T22:34:28.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:28 smithi162 conmon[39755]: audit 2024-01-15T22:34:27.855141+0000 mon.smithi099 (mon.0) 624 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:34:28.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:28 smithi162 conmon[39755]: audit 2024-01-15T22:34:27.855570+0000 mon.smithi099 (mon.0) 625 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-15T22:34:29.287 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:34:29.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:29 smithi099 conmon[35292]: cluster 2024-01-15T22:34:27.847996+0000 mgr.smithi099.cmjeyn (mgr.14184) 262 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:29.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:29 smithi099 conmon[35292]: cluster 2024-01-15T22:34:28.846037+0000 mon.smithi099 (mon.0) 626 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-15T22:34:29.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:29 smithi099 conmon[35292]: cluster 2024-01-15T22:34:28.846060+0000 mon.smithi099 (mon.0) 627 : cluster [INF] Cluster is now healthy 2024-01-15T22:34:29.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:29 smithi099 conmon[35292]: audit 2024-01-15T22:34:29.286067+0000 mon.smithi099 (mon.0) 628 : audit [DBG] from='client.? 172.21.15.99:0/2145541205' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:34:29.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:29 smithi162 conmon[39755]: cluster 2024-01-15T22:34:27.847996+0000 mgr.smithi099.cmjeyn (mgr.14184) 2024-01-15T22:34:29.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:29 smithi162 conmon[39755]: 262 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:29.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:29 smithi162 conmon[39755]: cluster 2024-01-15T22:34:28.846037+0000 mon.smithi099 (mon.0) 626 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-15T22:34:29.948 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:29 smithi162 conmon[39755]: cluster 2024-01-15T22:34:28.846060+0000 mon.smithi099 (mon.0) 627 : cluster [INF] Cluster is now healthy 2024-01-15T22:34:29.949 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:29 smithi162 conmon[39755]: audit 2024-01-15T22:34:29.286067+0000 mon.smithi099 (mon.0) 628 : audit [DBG] from='client.? 172.21.15.99:0/2145541205' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:34:30.195 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:34:31.196 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:34:31.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:31 smithi099 conmon[35292]: cluster 2024-01-15T22:34:29.848214+0000 mgr.smithi099.cmjeyn (mgr.14184) 263 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:31.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:31 smithi099 conmon[35292]: audit 2024-01-15T22:34:30.361557+0000 mon.smithi099 (mon.0) 629 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:34:31.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:31 smithi099 conmon[35292]: audit 2024-01-15T22:34:30.450784+0000 mon.smithi162 (mon.1) 18 : audit [INF] from='client.? 172.21.15.162:0/1051619796' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a17561fb-7c3a-4a81-9f90-2148e484efd3"}]: dispatch 2024-01-15T22:34:31.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:31 smithi099 conmon[35292]: audit 2024-01-15T22:34:30.451214+0000 mon.smithi099 (mon.0) 630 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a17561fb-7c3a-4a81-9f90-2148e484efd3"}]: dispatch 2024-01-15T22:34:31.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:31 smithi099 conmon[35292]: audit 2024-01-15T22:34:30.459004+0000 mon.smithi099 (mon.0) 631 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a17561fb-7c3a-4a81-9f90-2148e484efd3"}]': finished 2024-01-15T22:34:31.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:31 smithi099 conmon[35292]: cluster 2024-01-15T22:34:30.459045+0000 mon.smithi099 (mon.0) 632 : cluster [DBG] osdmap e34: 1 total, 0 up, 1 in 2024-01-15T22:34:31.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:31 smithi099 conmon[35292]: audit 2024-01-15T22:34:30.459139+0000 mon.smithi099 (mon.0) 633 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-15T22:34:31.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:31 smithi099 conmon[35292]: audit 2024-01-15T22:34:30.751808+0000 mon.smithi162 (mon.1) 19 : audit [INF] from='client.? 172.21.15.162:0/4197123884' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:34:31.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:31 smithi099 conmon[35292]: audit 2024-01-15T22:34:30.752190+0000 mon.smithi099 (mon.0) 634 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:34:31.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:31 smithi099 conmon[35292]: audit 2024-01-15T22:34:30.756659+0000 mon.smithi099 (mon.0) 635 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-15T22:34:31.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:31 smithi099 conmon[35292]: cluster 2024-01-15T22:34:30.756718+0000 mon.smithi099 (mon.0) 636 : cluster [DBG] osdmap e35: 0 total, 0 up, 0 in 2024-01-15T22:34:31.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:31 smithi162 conmon[39755]: cluster 2024-01-15T22:34:29.848214+0000 mgr.smithi099.cmjeyn (mgr.14184) 263 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:31.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:31 smithi162 conmon[39755]: audit 2024-01-15T22:34:30.361557+0000 mon.smithi099 (mon.0) 629 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:34:31.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:31 smithi162 conmon[39755]: audit 2024-01-15T22:34:30.450784+0000 mon.smithi162 (mon.1) 18 : audit [INF] from='client.? 172.21.15.162:0/1051619796' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a17561fb-7c3a-4a81-9f90-2148e484efd3"}]: dispatch 2024-01-15T22:34:31.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:31 smithi162 conmon[39755]: audit 2024-01-15T22:34:30.451214+0000 mon.smithi099 (mon.0) 630 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a17561fb-7c3a-4a81-9f90-2148e484efd3"}]: dispatch 2024-01-15T22:34:31.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:31 smithi162 conmon[39755]: audit 2024-01-15T22:34:30.459004+0000 mon.smithi099 (mon.0) 631 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a17561fb-7c3a-4a81-9f90-2148e484efd3"}]': finished 2024-01-15T22:34:31.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:31 smithi162 conmon[39755]: cluster 2024-01-15T22:34:30.459045+0000 mon.smithi099 (mon.0) 632 : cluster [DBG] osdmap e34: 1 total, 0 up, 1 in 2024-01-15T22:34:31.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:31 smithi162 conmon[39755]: audit 2024-01-15T22:34:30.459139+0000 mon.smithi099 (mon.0) 633 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-15T22:34:31.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:31 smithi162 conmon[39755]: audit 2024-01-15T22:34:30.751808+0000 mon.smithi162 (mon.1) 19 : audit [INF] from='client.? 172.21.15.162:0/4197123884' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:34:31.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:31 smithi162 conmon[39755]: audit 2024- 2024-01-15T22:34:31.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:31 smithi162 conmon[39755]: 01-15T22:34:30.752190+0000 mon.smithi099 (mon.0) 634 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:34:31.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:31 smithi162 conmon[39755]: audit 2024-01-15T22:34:30.756659+0000 mon.smithi099 (mon.0) 635 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-15T22:34:31.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:31 smithi162 conmon[39755]: cluster 2024-01-15T22:34:30.756718+0000 mon.smithi099 (mon.0) 636 : cluster [DBG] osdmap e35: 0 total, 0 up, 0 in 2024-01-15T22:34:32.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: cephadm 2024-01-15T22:34:31.465711+0000 mgr.smithi099.cmjeyn (mgr.14184) 2024-01-15T22:34:32.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: 264 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi162', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-15T22:34:32.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:34:32.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:34:32.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: return f(*arg) 2024-01-15T22:34:32.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:34:32.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:34:32.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:34:32.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: code, '\n'.join(err))) 2024-01-15T22:34:32.657 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:34:32.657 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsz9jh9cg:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpa_l90r3k:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:34:32.657 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:34:32.657 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:34:32.657 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:34:32.657 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new a17561fb-7c3a-4a81-9f90-2148e484efd3 2024-01-15T22:34:32.657 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:34:32.657 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:34:32.657 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:34:32.657 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:34:32.657 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:34:32.657 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr self.prepare() 2024-01-15T22:34:32.657 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:34:32.657 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:34:32.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:34:32.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:34:32.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:34:32.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:34:32.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:34:32.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:34:32.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:34:32.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:34:32.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:34:32.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:34:32.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:34:32.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:34:32.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:34:32.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:34:32.659 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:34:32.659 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:34:32.659 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:34:32.659 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:34:32.659 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:34:32.659 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:34:32.659 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:34:32.659 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:34:32.659 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:34:32.659 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:34:32.659 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:34:32.659 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:34:32.659 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:34:32.659 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:34:32.659 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:34:32.660 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:34:32.660 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:34:32.660 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:34:32.660 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:34:32.660 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:34:32.660 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:34:32.660 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:34:32.660 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:34:32.660 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:34:32.660 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:34:32.660 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:34:32.660 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:34:32.660 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:34:32.660 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:34:32.660 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:34:32.661 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:34:32.661 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:34:32.661 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:34:32.661 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:34:32.661 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:34:32.661 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:34:32.661 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: main() 2024-01-15T22:34:32.661 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:34:32.661 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: r = ctx.func(ctx) 2024-01-15T22:34:32.661 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:34:32.661 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:34:32.661 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:34:32.661 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:34:32.661 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:34:32.662 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:34:32.662 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fd 2024-01-15T22:34:32.662 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: a83f90d2", line 1927, in _validate_fsid 2024-01-15T22:34:32.662 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:34:32.662 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:34:32.662 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:34:32.662 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:34:32.662 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:34:32.662 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsz9jh9cg:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpa_l90r3k:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:34:32.662 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: audit 2024-01-15T22:34:31.504829+0000 mon.smithi099 (mon.0) 637 : audit [INF] from='client.? 172.21.15.99:0/895283832' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80052a39-3112-4a2d-82fd-f2824b4a0701"}]: dispatch 2024-01-15T22:34:32.662 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: audit 2024-01-15T22:34:31.508369+0000 mon.smithi099 (mon.0) 638 : audit [INF] from='client.? 172.21.15.99:0/895283832' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80052a39-3112-4a2d-82fd-f2824b4a0701"}]': finished 2024-01-15T22:34:32.662 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: cluster 2024-01-15T22:34:31.508408+0000 mon.smithi099 (mon.0) 639 : cluster [DBG] osdmap e36: 1 total, 0 up, 1 in 2024-01-15T22:34:32.662 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: audit 2024-01-15T22:34:31.508511+0000 mon.smithi099 (mon.0) 640 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-15T22:34:32.662 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: audit 2024-01-15T22:34:31.813972+0000 mon.smithi099 (mon.0) 641 : audit [INF] from='client.? 172.21.15.99:0/1460903766' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:34:32.662 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: audit 2024-01-15T22:34:31.817465+0000 mon.smithi099 (mon.0) 642 : audit [INF] from='client.? 172.21.15.99:0/1460903766' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-15T22:34:32.663 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:32 smithi099 conmon[35292]: cluster 2024-01-15T22:34:31.817504+0000 mon.smithi099 (mon.0) 643 : cluster [DBG] osdmap e37: 0 total, 0 up, 0 in 2024-01-15T22:34:32.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: cephadm 2024-01-15T22:34:31.465711+0000 mgr.smithi099.cmjeyn (mgr.14184) 264 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi162', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-15T22:34:32.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:34:32.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:34:32.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: return f(*arg) 2024-01-15T22:34:32.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:34:32.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:34:32.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:34:32.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: code, '\n'.join(err))) 2024-01-15T22:34:32.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:34:32.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsz9jh9cg:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpa_l90r3k:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:34:32.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:34:32.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:34:32.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:34:32.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new a17561fb-7c3a-4a81-9f90-2148e484efd3 2024-01-15T22:34:32.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:34:32.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:34:32.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:34:32.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:34:32.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:34:32.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr self.prepare() 2024-01-15T22:34:32.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:34:32.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:34:32.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:34:32.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:34:32.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:34:32.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:34:32.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:34:32.700 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:34:32.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:34:32.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:34:32.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:34:32.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:34:32.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:34:32.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:34:32.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:34:32.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:34:32.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:34:32.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:34:32.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:34:32.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:34:32.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:34:32.701 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:34:32.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:34:32.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:34:32.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:34:32.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:34:32.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:34:32.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:34:32.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:34:32.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:34:32.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:34:32.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:34:32.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:34:32.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:34:32.702 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:34:32.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:34:32.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:34:32.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:34:32.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:34:32.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:34:32.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:34:32.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:34:32.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:34:32.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:34:32.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:34:32.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:34:32.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:34:32.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:34:32.703 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:34:32.704 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:34:32.704 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:34:32.704 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:34:32.704 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: main() 2024-01-15T22:34:32.704 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:34:32.704 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: r = ctx.func(ctx) 2024-01-15T22:34:32.704 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:34:32.704 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:34:32.704 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:34:32.705 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:34:32.705 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:34:32.705 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:34:32.705 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-8777 2024-01-15T22:34:32.705 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: 4f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:34:32.705 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:34:32.705 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:34:32.705 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:34:32.705 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:34:32.705 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:34:32.705 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsz9jh9cg:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpa_l90r3k:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:34:32.705 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: audit 2024-01-15T22:34:31.504829+0000 mon.smithi099 (mon.0) 637 : audit [INF] from='client.? 172.21.15.99:0/895283832' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80052a39-3112-4a2d-82fd-f2824b4a0701"}]: dispatch 2024-01-15T22:34:32.705 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: audit 2024-01-15T22:34:31.508369+0000 mon.smithi099 (mon.0) 638 : audit [INF] from='client.? 172.21.15.99:0/895283832' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80052a39-3112-4a2d-82fd-f2824b4a0701"}]': finished 2024-01-15T22:34:32.705 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: cluster 2024-01-15T22:34:31.508408+0000 mon.smithi099 (mon.0) 639 : cluster [DBG] osdmap e36: 1 total, 0 up, 1 in 2024-01-15T22:34:32.706 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: audit 2024-01-15T22:34:31.508511+0000 mon.smithi099 (mon.0) 640 : audit [DBG] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-15T22:34:32.706 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: audit 2024-01-15T22:34:31.813972+0000 mon.smithi099 (mon.0) 641 : audit [INF] from='client.? 172.21.15.99:0/1460903766' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-15T22:34:32.706 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: audit 2024-01-15T22:34:31.817465+0000 mon.smithi099 (mon.0) 642 : audit [INF] from='client.? 172.21.15.99:0/1460903766' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-15T22:34:32.706 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:32 smithi162 conmon[39755]: cluster 2024-01-15T22:34:31.817504+0000 mon.smithi099 (mon.0) 643 : cluster [DBG] osdmap e37: 0 total, 0 up, 0 in 2024-01-15T22:34:33.414 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:34:33.967 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":37,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:34:34.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:33 smithi099 conmon[35292]: cluster 2024-01-15T22:34:31.848433+0000 mgr.smithi099.cmjeyn (mgr.14184) 265 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:34.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:33 smithi099 conmon[35292]: audit 2024-01-15T22:34:33.413542+0000 mon.smithi099 (mon.0) 644 : audit [DBG] from='client.? 172.21.15.99:0/309271558' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:34:34.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:33 smithi162 conmon[39755]: cluster 2024-01-15T22:34:31.848433+0000 mgr.smithi099.cmjeyn (mgr.14184) 265 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:34.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:33 smithi162 conmon[39755]: audit 2024-01-15T22:34:33.413542+0000 mon.smithi099 (mon.0) 644 : audit [DBG] 2024-01-15T22:34:34.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:33 smithi162 conmon[39755]: from='client.? 172.21.15.99:0/309271558' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:34:34.968 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:34:35.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: cephadm 2024-01-15T22:34:32.877686+0000 mgr.smithi099.cmjeyn (mgr.14184) 266 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi162', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-15T22:34:35.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:34:35.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:34:35.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: return f(*arg) 2024-01-15T22:34:35.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:34:35.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:34:35.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:34:35.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: code, '\n'.join(err))) 2024-01-15T22:34:35.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi099/config 2024-01-15T22:34:35.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp3mtzk_cd:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpqc6sog24:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:34:35.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:34:35.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:34:35.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:34:35.156 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 80052a39-3112-4a2d-82fd-f2824b4a0701 2024-01-15T22:34:35.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:34:35.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:34:35.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:34:35.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:34:35.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:34:35.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr self.prepare() 2024-01-15T22:34:35.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:34:35.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:34:35.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:34:35.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:34:35.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:34:35.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:34:35.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:34:35.157 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:34:35.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:34:35.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:34:35.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:34:35.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:34:35.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:34:35.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:34:35.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:34:35.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:34:35.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:34:35.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:34:35.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:34:35.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:34:35.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:34:35.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:34:35.158 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:34:35.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:34:35.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:34:35.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:34:35.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:34:35.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:34:35.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:34:35.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:34:35.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:34:35.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:34:35.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:34:35.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:34:35.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:34:35.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:34:35.159 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:34:35.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:34:35.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:34:35.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:34:35.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:34:35.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:34:35.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:34:35.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:34:35.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:34:35.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:34:35.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:34:35.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:34:35.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:34:35.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:34:35.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:34:35.160 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:34:35.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: main() 2024-01-15T22:34:35.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:34:35.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: r = ctx.func(ctx) 2024-01-15T22:34:35.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:34:35.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:34:35.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:34:35.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:34:35.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:34:35.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:34:35.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-8777 2024-01-15T22:34:35.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: 4f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:34:35.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:34:35.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:34:35.161 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:34:35.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:34:35.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:34:35.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp3mtzk_cd:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpqc6sog24:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:34:35.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: cephadm 2024-01-15T22:34:32.879290+0000 mgr.smithi099.cmjeyn (mgr.14184) 267 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-15T22:34:35.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: service_id: all-available-devices 2024-01-15T22:34:35.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: service_name: osd.all-available-devices 2024-01-15T22:34:35.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: placement: 2024-01-15T22:34:35.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: host_pattern: '*' 2024-01-15T22:34:35.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: spec: 2024-01-15T22:34:35.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: data_devices: 2024-01-15T22:34:35.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: all: true 2024-01-15T22:34:35.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: filter_logic: AND 2024-01-15T22:34:35.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: objectstore: bluestore 2024-01-15T22:34:35.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:34:35.162 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsz9jh9cg:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpa_l90r3k:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:34:35.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:34:35.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:34:35.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:34:35.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new a17561fb-7c3a-4a81-9f90-2148e484efd3 2024-01-15T22:34:35.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:34:35.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:34:35.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:34:35.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:34:35.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:34:35.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr self.prepare() 2024-01-15T22:34:35.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:34:35.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:34:35.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:34:35.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:34:35.163 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:34:35.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:34:35.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:34:35.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:34:35.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:34:35.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:34:35.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:34:35.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:34:35.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:34:35.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:34:35.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:34:35.164 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:34:35.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:34:35.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:34:35.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:34:35.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:34:35.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:34:35.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:34:35.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:34:35.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:34:35.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:34:35.165 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:34:35.166 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:34:35.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:34:35.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:34:35.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:34:35.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:34:35.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:34:35.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:34:35.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:34:35.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:34:35.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:34:35.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:34:35.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:34:35.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:34:35.167 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:34:35.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:34:35.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:34:35.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:34:35.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:34:35.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:34:35.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/ 2024-01-15T22:34:35.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: lvm/zap.py", line 302, in zap_osd 2024-01-15T22:34:35.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:34:35.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:34:35.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:34:35.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:34:35.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:34:35.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:34:35.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: main() 2024-01-15T22:34:35.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:34:35.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: r = ctx.func(ctx) 2024-01-15T22:34:35.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:34:35.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:34:35.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:34:35.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:34:35.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:34:35.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:34:35.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:34:35.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:34:35.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:34:35.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:34:35.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:34:35.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:34:35.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsz9jh9cg:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpa_l90r3k:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:34:35.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:34:35.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-15T22:34:35.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: if self._apply_service(spec): 2024-01-15T22:34:35.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-15T22:34:35.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-15T22:34:35.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-15T22:34:35.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-15T22:34:35.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-15T22:34:35.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-15T22:34:35.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-15T22:34:35.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-15T22:34:35.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-15T22:34:35.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: raise self._value 2024-01-15T22:34:35.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-15T22:34:35.170 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: result = (True, func(*args, **kwds)) 2024-01-15T22:34:35.171 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-15T22:34:35.171 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: return list(map(*args)) 2024-01-15T22:34:35.171 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:34:35.171 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: return f(*arg) 2024-01-15T22:34:35.171 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:34:35.171 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:34:35.171 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:34:35.171 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: code, '\n'.join(err))) 2024-01-15T22:34:35.171 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:34:35.171 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsz9jh9cg:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpa_l90r3k:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:34:35.171 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:34:35.171 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:34:35.171 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:34:35.171 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new a17561fb-7c3a-4a81-9f90-2148e484efd3 2024-01-15T22:34:35.171 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:34:35.172 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:34:35.172 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:34:35.172 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:34:35.172 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:34:35.172 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr self.prepare() 2024-01-15T22:34:35.172 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:34:35.172 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:34:35.172 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:34:35.172 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:34:35.172 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:34:35.172 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:34:35.172 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:34:35.172 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:34:35.172 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:34:35.173 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:34:35.173 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:34:35.173 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:34:35.173 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:34:35.173 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:34:35.173 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:34:35.173 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:34:35.173 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:34:35.173 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr 2024-01-15T22:34:35.173 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:34:35.173 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:34:35.173 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:34:35.173 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:34:35.173 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:34:35.173 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:34:35.173 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:34:35.174 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:34:35.174 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:34:35.174 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr instance.main() 2024-01-15T22:34:35.174 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:34:35.174 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:34:35.174 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:34:35.174 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:34:35.174 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:34:35.174 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:34:35.174 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:34:35.174 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:34:35.174 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:34:35.174 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:34:35.174 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:34:35.174 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:34:35.175 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:34:35.175 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:34:35.175 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:34:35.175 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:34:35.175 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:34:35.175 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:34:35.175 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:34:35.175 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:34:35.175 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:34:35.175 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:34:35.175 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:34:35.175 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: Traceback (most recent call last): 2024-01-15T22:34:35.175 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:34:35.175 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: main() 2024-01-15T22:34:35.176 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:34:35.176 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: r = ctx.func(ctx) 2024-01-15T22:34:35.176 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:34:35.176 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:34:35.176 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:34:35.176 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:34:35.176 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:34:35.176 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:34:35.176 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:34:35.176 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: return func(ctx) 2024-01-15T22:34:35.176 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:34:35.176 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:34:35.176 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:34:35.177 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:34:35.177 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsz9jh9cg:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpa_l90r3k:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:34:35.177 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: cluster 2024-01-15T22:34:32.880022+0000 mgr.smithi099.cmjeyn (mgr.14184) 268 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:35.177 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:34 smithi099 conmon[35292]: cluster 2024-01-15T22:34:33.816981+0000 mon.smithi099 (mon.0) 645 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-15T22:34:35.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: cephadm 2024-01-15T22:34:32.877686+0000 mgr.smithi099.cmjeyn (mgr.14184) 266 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi162', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-15T22:34:35.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:34:35.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:34:35.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: return f(*arg) 2024-01-15T22:34:35.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:34:35.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:34:35.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:34:35.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: code, '\n'.join(err))) 2024-01-15T22:34:35.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi099/config 2024-01-15T22:34:35.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp3mtzk_cd:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpqc6sog24:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:34:35.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:34:35.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:34:35.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:34:35.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 80052a39-3112-4a2d-82fd-f2824b4a0701 2024-01-15T22:34:35.199 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:34:35.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:34:35.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:34:35.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:34:35.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:34:35.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr self.prepare() 2024-01-15T22:34:35.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:34:35.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:34:35.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:34:35.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:34:35.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:34:35.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:34:35.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:34:35.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:34:35.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:34:35.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:34:35.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:34:35.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:34:35.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:34:35.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:34:35.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:34:35.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:34:35.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:34:35.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:34:35.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:34:35.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:34:35.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:34:35.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:34:35.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:34:35.201 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:34:35.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:34:35.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:34:35.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:34:35.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:34:35.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:34:35.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:34:35.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:34:35.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:34:35.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:34:35.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:34:35.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:34:35.202 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:34:35.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:34:35.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:34:35.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:34:35.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:34:35.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:34:35.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:34:35.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:34:35.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:34:35.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:34:35.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:34:35.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:34:35.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:34:35.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:34:35.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:34:35.203 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:34:35.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:34:35.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: main() 2024-01-15T22:34:35.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:34:35.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: r = ctx.func(ctx) 2024-01-15T22:34:35.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:34:35.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:34:35.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:34:35.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:34:35.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:34:35.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:34:35.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-8777 2024-01-15T22:34:35.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: 4f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:34:35.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:34:35.204 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:34:35.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:34:35.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:34:35.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:34:35.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp3mtzk_cd:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpqc6sog24:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:34:35.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: cephadm 2024-01-15T22:34:32.879290+0000 mgr.smithi099.cmjeyn (mgr.14184) 267 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-15T22:34:35.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: service_id: all-available-devices 2024-01-15T22:34:35.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: service_name: osd.all-available-devices 2024-01-15T22:34:35.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: placement: 2024-01-15T22:34:35.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: host_pattern: '*' 2024-01-15T22:34:35.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: spec: 2024-01-15T22:34:35.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: data_devices: 2024-01-15T22:34:35.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: all: true 2024-01-15T22:34:35.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: filter_logic: AND 2024-01-15T22:34:35.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: objectstore: bluestore 2024-01-15T22:34:35.205 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:34:35.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsz9jh9cg:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpa_l90r3k:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:34:35.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:34:35.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:34:35.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:34:35.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new a17561fb-7c3a-4a81-9f90-2148e484efd3 2024-01-15T22:34:35.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:34:35.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:34:35.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:34:35.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:34:35.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:34:35.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr self.prepare() 2024-01-15T22:34:35.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:34:35.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:34:35.206 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:34:35.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:34:35.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:34:35.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:34:35.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:34:35.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:34:35.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:34:35.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:34:35.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:34:35.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:34:35.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:34:35.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:34:35.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:34:35.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:34:35.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:34:35.207 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:34:35.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:34:35.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:34:35.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:34:35.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:34:35.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:34:35.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:34:35.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:34:35.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:34:35.208 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:34:35.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:34:35.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:34:35.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:34:35.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:34:35.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:34:35.209 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:34:35.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:34:35.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:34:35.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:34:35.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:34:35.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:34:35.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:34:35.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:34:35.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:34:35.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:34:35.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:34:35.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:34:35.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:34:35.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/ 2024-01-15T22:34:35.210 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: lvm/zap.py", line 302, in zap_osd 2024-01-15T22:34:35.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:34:35.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:34:35.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:34:35.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:34:35.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:34:35.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:34:35.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: main() 2024-01-15T22:34:35.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:34:35.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: r = ctx.func(ctx) 2024-01-15T22:34:35.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:34:35.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:34:35.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:34:35.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:34:35.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:34:35.211 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:34:35.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:34:35.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:34:35.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:34:35.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:34:35.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:34:35.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:34:35.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsz9jh9cg:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpa_l90r3k:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:34:35.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:34:35.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-15T22:34:35.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: if self._apply_service(spec): 2024-01-15T22:34:35.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-15T22:34:35.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-15T22:34:35.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-15T22:34:35.212 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-15T22:34:35.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-15T22:34:35.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-15T22:34:35.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-15T22:34:35.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-15T22:34:35.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-15T22:34:35.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: raise self._value 2024-01-15T22:34:35.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-15T22:34:35.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: result = (True, func(*args, **kwds)) 2024-01-15T22:34:35.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-15T22:34:35.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: return list(map(*args)) 2024-01-15T22:34:35.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-15T22:34:35.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: return f(*arg) 2024-01-15T22:34:35.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-15T22:34:35.213 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-15T22:34:35.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-15T22:34:35.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: code, '\n'.join(err))) 2024-01-15T22:34:35.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/mon.smithi162/config 2024-01-15T22:34:35.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsz9jh9cg:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpa_l90r3k:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:34:35.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-15T22:34:35.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr --> relative data size: 1.0 2024-01-15T22:34:35.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-15T22:34:35.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new a17561fb-7c3a-4a81-9f90-2148e484efd3 2024-01-15T22:34:35.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-15T22:34:35.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-15T22:34:35.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr stderr: purged osd.0 2024-01-15T22:34:35.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:34:35.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-15T22:34:35.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr self.prepare() 2024-01-15T22:34:35.214 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:34:35.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:34:35.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-15T22:34:35.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-15T22:34:35.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-15T22:34:35.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-15T22:34:35.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-15T22:34:35.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr partitions = get_partitions() 2024-01-15T22:34:35.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-15T22:34:35.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:34:35.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-15T22:34:35.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:34:35.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr Traceback (most recent call last): 2024-01-15T22:34:35.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-15T22:34:35.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-15T22:34:35.215 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-15T22:34:35.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr self.main(self.argv) 2024-01-15T22:34:35.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr 2024-01-15T22:34:35.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-15T22:34:35.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr return f(*a, **kw) 2024-01-15T22:34:35.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-15T22:34:35.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-15T22:34:35.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:34:35.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:34:35.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-15T22:34:35.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-15T22:34:35.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-15T22:34:35.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr instance.main() 2024-01-15T22:34:35.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:34:35.216 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:34:35.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-15T22:34:35.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr self._execute(plan) 2024-01-15T22:34:35.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-15T22:34:35.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-15T22:34:35.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:34:35.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:34:35.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-15T22:34:35.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-15T22:34:35.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-15T22:34:35.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-15T22:34:35.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-15T22:34:35.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-15T22:34:35.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-15T22:34:35.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr self.zap_osd() 2024-01-15T22:34:35.217 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-15T22:34:35.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr return func(*a, **kw) 2024-01-15T22:34:35.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-15T22:34:35.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-15T22:34:35.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-15T22:34:35.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-15T22:34:35.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-15T22:34:35.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: Traceback (most recent call last): 2024-01-15T22:34:35.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-15T22:34:35.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: main() 2024-01-15T22:34:35.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-15T22:34:35.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: r = ctx.func(ctx) 2024-01-15T22:34:35.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-15T22:34:35.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:34:35.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-15T22:34:35.218 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:34:35.219 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-15T22:34:35.219 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:34:35.219 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-15T22:34:35.219 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: return func(ctx) 2024-01-15T22:34:35.219 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-15T22:34:35.219 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-15T22:34:35.219 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: File "/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-15T22:34:35.219 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-15T22:34:35.219 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi162 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/run/ceph:z -v /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsz9jh9cg:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpa_l90r3k:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-15T22:34:35.219 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: cluster 2024-01-15T22:34:32.880022+0000 mgr.smithi099.cmjeyn (mgr.14184) 268 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:35.219 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:34 smithi162 conmon[39755]: cluster 2024-01-15T22:34:33.816981+0000 mon.smithi099 (mon.0) 645 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-15T22:34:36.764 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:34:37.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:36 smithi099 conmon[35292]: cluster 2024-01-15T22:34:34.880665+0000 mgr.smithi099.cmjeyn (mgr.14184) 269 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:37.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:36 smithi099 conmon[35292]: audit 2024-01-15T22:34:36.762816+0000 mon.smithi099 (mon.0) 646 : audit [DBG] from='client.? 172.21.15.99:0/2922317645' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:34:37.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:36 smithi162 conmon[39755]: cluster 2024-01-15T22:34:34.880665+0000 mgr.smithi099.cmjeyn (mgr.14184) 269 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:37.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:36 smithi162 conmon[39755]: audit 2024-01-15T22:34:36 2024-01-15T22:34:37.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:36 smithi162 conmon[39755]: .762816+0000 mon.smithi099 (mon.0) 646 : audit [DBG] from='client.? 172.21.15.99:0/2922317645' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:34:37.282 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":37,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:34:38.283 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:34:39.083 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:38 smithi099 conmon[35292]: cluster 2024-01-15T22:34:36.881182+0000 2024-01-15T22:34:39.083 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:38 smithi099 conmon[35292]: mgr.smithi099.cmjeyn (mgr.14184) 270 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:39.200 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:38 smithi162 conmon[39755]: cluster 2024-01-15T22:34:36.881182+0000 mgr.smithi099.cmjeyn (mgr.14184) 270 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:40.024 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:34:40.583 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":37,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:34:41.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:40 smithi099 conmon[35292]: cluster 2024-01-15T22:34:38.881605+0000 mgr.smithi099.cmjeyn (mgr.14184) 271 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:41.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:40 smithi099 conmon[35292]: audit 2024-01-15T22:34:40.022956+0000 mon.smithi099 (mon.0) 647 : audit [DBG] from='client.? 172.21.15.99:0/2604299600' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:34:41.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:40 smithi162 conmon[39755]: cluster 2024-01-15T22:34:38.881605+0000 mgr.smithi099.cmjeyn (mgr.14184) 271 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:41.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:40 smithi162 conmon[39755]: audit 2024-01-15T22:34:40.022956+0000 mon.smithi099 (mon.0) 647 : audit [DBG] from='client.? 172.21.15.99:0/2604299600' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:34:41.584 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:34:43.012 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:42 smithi099 conmon[35292]: cluster 2024-01-15T22:34:40.882059 2024-01-15T22:34:43.012 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:42 smithi099 conmon[35292]: +0000 mgr.smithi099.cmjeyn (mgr.14184) 272 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:43.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:42 smithi162 conmon[39755]: cluster 2024-01-15T22:34:40.882059+0000 mgr.smithi099.cmjeyn (mgr.14184) 272 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:43.320 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:34:43.898 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":37,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:34:44.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:43 smithi099 conmon[35292]: cluster 2024-01-15T22:34:42.882427+0000 mgr.smithi099.cmjeyn (mgr.14184) 273 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:44.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:43 smithi099 conmon[35292]: audit 2024-01-15T22:34:43.319653+0000 mon.smithi099 (mon.0) 648 : audit [DBG] from='client.? 172.21.15.99:0/2024295977' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:34:44.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:43 smithi162 conmon[39755]: cluster 2024-01-15T22:34:42.882427+0000 mgr.smithi099.cmjeyn (mgr.14184) 273 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:44.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:43 smithi162 conmon[39755]: audit 2024-01-15T22:34:44.198 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:43 smithi162 conmon[39755]: 2024-01-15T22:34:43.319653+0000 mon.smithi099 (mon.0) 648 : audit [DBG] from='client.? 172.21.15.99:0/2024295977' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:34:44.899 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:34:46.617 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:34:46.644 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:46 smithi099 conmon[35292]: cluster 2024-01-15T22:34:44.883036+0000 mgr.smithi099.cmjeyn (mgr.14184) 274 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:46.644 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:46 smithi099 conmon[35292]: audit 2024-01-15T22:34:45.361367+0000 mon.smithi099 (mon.0) 649 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:34:46.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:46 smithi162 conmon[39755]: cluster 2024-01-15T22:34:44.883036+0000 mgr.smithi099.cmjeyn (mgr.14184) 274 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:46.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:46 smithi162 conmon[39755]: audit 2024-01-15T22:34:45.361367+0000 mon.smithi099 (mon.0) 649 : audit [INF] from='mgr.14184 172.21.15.99:0/1302649713' entity='mgr.smithi099.cmjeyn' 2024-01-15T22:34:47.197 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":37,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:34:47.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:47 smithi099 conmon[35292]: audit 2024-01-15T22:34:46.616533+0000 mon.smithi099 (mon.0) 650 : audit [DBG] from='client.? 172.21.15.99:0/2555757341' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:34:47.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:47 smithi162 conmon[39755]: audit 2024-01-15T22:34:46.616533+0000 mon.smithi099 (mon.0) 650 : audit [DBG] from='client.? 172.21.15.99:0/2555757341' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:34:48.199 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:34:48.647 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:48 smithi099 conmon[35292]: cluster 2024-01-15T22:34:46.883592+0000 mgr.smithi099.cmjeyn (mgr.14184) 275 : cluster [DBG] 2024-01-15T22:34:48.647 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:48 smithi099 conmon[35292]: pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:48.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:48 smithi162 conmon[39755]: cluster 2024-01-15T22:34:46.883592+0000 mgr.smithi099.cmjeyn (mgr.14184) 275 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:49.918 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:34:50.523 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":37,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:34:50.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:50 smithi099 conmon[35292]: cluster 2024-01-15T22:34:48.884184+0000 mgr.smithi099.cmjeyn (mgr.14184) 276 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:50.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:50 smithi099 conmon[35292]: audit 2024-01-15T22:34:49.918263+0000 mon.smithi099 (mon.0) 651 : audit [DBG] from='client.? 172.21.15.99:0/4132558386' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:34:50.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:50 smithi162 conmon[39755]: cluster 2024-01-15T22:34:48.884184+0000 mgr.smithi099.cmjeyn (mgr.14184) 276 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:50.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:50 smithi162 conmon[39755]: audit 2024-01-15T22:34:49.918263+0000 mon.smithi099 (mon.0) 651 : audit [DBG] from='client.? 172.21.15.99:0/4132558386' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:34:51.525 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:34:52.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:52 smithi099 conmon[35292]: cluster 2024-01-15T22:34:50.884674+0000 mgr.smithi099.cmjeyn (mgr.14184) 277 : 2024-01-15T22:34:52.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:52 smithi099 conmon[35292]: cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:52.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:52 smithi162 conmon[39755]: cluster 2024-01-15T22:34:50.884674+0000 mgr.smithi099.cmjeyn (mgr.14184) 277 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:53.332 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:34:53.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:53 smithi099 conmon[35292]: audit 2024-01-15T22:34:53. 2024-01-15T22:34:53.656 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:53 smithi099 conmon[35292]: 331815+0000 mon.smithi099 (mon.0) 652 : audit [DBG] from='client.? 172.21.15.99:0/1739444528' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:34:53.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:53 smithi162 conmon[39755]: audit 2024-01-15T22:34:53.331815+0000 mon.smithi099 (mon.0) 652 : audit [DBG] from='client.? 172.21.15.99:0/1739444528' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:34:53.895 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":37,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:34:54.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:54 smithi099 conmon[35292]: cluster 2024-01-15T22:34:52.885068+0000 mgr.smithi099.cmjeyn (mgr.14184) 278 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:54.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:54 smithi162 conmon[39755]: cluster 2024-01-15T22:34:52.885068+0000 mgr.smithi099.cmjeyn (mgr.14184) 278 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:54.896 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:34:56.645 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:34:56.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:56 smithi099 conmon[35292]: cluster 2024-01-15T22:34:54.885491+0000 mgr.smithi099.cmjeyn (mgr.14184) 279 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:56.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:56 smithi162 conmon[39755]: cluster 2024-01-15T22:34:54.885491+0000 mgr.smithi099.cmjeyn (mgr.14184) 279 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:57.186 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":37,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:34:57.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:57 smithi099 conmon[35292]: audit 2024-01-15T22:34:56.643473+0000 mon.smithi099 (mon.0) 653 : audit 2024-01-15T22:34:57.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:57 smithi099 conmon[35292]: [DBG] from='client.? 172.21.15.99:0/1081476610' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:34:57.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:57 smithi162 conmon[39755]: audit 2024-01-15T22:34:56.643473+0000 mon.smithi099 (mon.0) 653 : audit [DBG] from='client.? 172.21.15.99:0/1081476610' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:34:58.188 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:34:58.628 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:58 smithi099 conmon[35292]: cluster 2024-01-15T22:34:56.886159+0000 mgr.smithi099.cmjeyn (mgr. 2024-01-15T22:34:58.628 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:34:58 smithi099 conmon[35292]: 14184) 280 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:58.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:34:58 smithi162 conmon[39755]: cluster 2024-01-15T22:34:56.886159+0000 mgr.smithi099.cmjeyn (mgr.14184) 280 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:34:59.957 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:35:00.517 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":37,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:35:00.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:35:00 smithi162 conmon[39755]: cluster 2024-01-15T22:34:58.886763+0000 mgr.smithi099.cmjeyn (mgr.14184) 281 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:35:00.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:35:00 smithi162 conmon[39755]: audit 2024-01-15T22:34:59.955225+0000 mon.smithi099 (mon.0) 654 : audit 2024-01-15T22:35:00.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:35:00 smithi162 conmon[39755]: [DBG] from='client.? 172.21.15.99:0/2995175988' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:35:00.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:35:00 smithi099 conmon[35292]: cluster 2024-01-15T22:34:58.886763+0000 mgr.smithi099.cmjeyn (mgr.14184) 281 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:35:00.905 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:35:00 smithi099 conmon[35292]: audit 2024-01-15T22:34:59.955225+0000 mon.smithi099 (mon.0) 654 : audit [DBG] from='client.? 172.21.15.99:0/2995175988' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:35:01.519 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 -- ceph osd stat -f json 2024-01-15T22:35:02.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:35:02 smithi099 conmon[35292]: cluster 2024-01-15T22:35:00.887184+0000 mgr.smithi099.cmjeyn (mgr.14184) 282 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:35:02.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:35:02 smithi099 conmon[35292]: 2024-01-15T22:35:02.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:35:02 smithi162 conmon[39755]: cluster 2024-01-15T22:35:00.887184+0000 mgr.smithi099.cmjeyn (mgr.14184) 282 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-15T22:35:03.325 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-15T22:35:03.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:35:03 smithi099 conmon[35292]: audit 2024-01-15T22:35:03.324315+0000 mon.smithi099 (mon.0) 655 : audit [DBG] from='client.? 172.21.15.99:0/1381992095' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:35:03.699 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:35:03 smithi162 conmon[39755]: audit 2024-01-15T22:35:03.324315+0000 mon.smithi099 (mon.0) 655 : audit [DBG] from='client.? 172.21.15.99:0/1381992095' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-15T22:35:03.910 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":37,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705357709,"num_remapped_pgs":0} 2024-01-15T22:35:03.911 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 779, in ceph_osds while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (121) after waiting for 120 seconds 2024-01-15T22:35:03.912 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-15T22:35:03.944 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-15T22:35:03.998 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-01-15T22:35:03.998 DEBUG:teuthology.orchestra.run.smithi099:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-15T22:35:04.015 DEBUG:teuthology.orchestra.run.smithi162:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-15T22:35:04.035 INFO:tasks.cephadm:Stopping all daemons... 2024-01-15T22:35:04.036 INFO:tasks.cephadm.mon.smithi099:Stopping mon.smithi099... 2024-01-15T22:35:04.036 DEBUG:teuthology.orchestra.run.smithi099:> sudo systemctl stop ceph-eb17a8d4-b3f4-11ee-95ad-87774f69a715@mon.smithi099 2024-01-15T22:35:04.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:35:04 smithi099 systemd[1]: Stopping Ceph mon.smithi099 for eb17a8d4-b3f4-11ee-95ad-87774f69a715... 2024-01-15T22:35:04.155 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:35:04 smithi099 bash[99180]: Error: no container with name or ID ceph-eb17a8d4-b3f4-11ee-95ad-87774f69a715-mon.smithi099 found: no such container 2024-01-15T22:35:04.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:35:04 smithi099 conmon[35292]: debug 2024-01-15T22:35:04.351+0000 7f2cf098c700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.smithi099 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-15T22:35:04.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:35:04 smithi099 conmon[35292]: debug 2024-01-15T22:35:04.351+0000 7f2cf098c700 -1 mon.smithi099@0(leader) e2 *** Got Signal Terminated *** 2024-01-15T22:35:04.655 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 15 22:35:04 smithi099 bash[99298]: 8d59a446f27314a7322e7a61fb8d7f434b971fb59ed6d2f67dc9671662edc7eb 2024-01-15T22:35:05.048 DEBUG:teuthology.orchestra.run.smithi099:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb17a8d4-b3f4-11ee-95ad-87774f69a715@mon.smithi099.service' 2024-01-15T22:35:05.121 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-15T22:35:05.122 INFO:tasks.cephadm.mon.smithi099:Stopped mon.smithi099 2024-01-15T22:35:05.122 INFO:tasks.cephadm.mon.smithi162:Stopping mon.smithi162... 2024-01-15T22:35:05.122 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-eb17a8d4-b3f4-11ee-95ad-87774f69a715@mon.smithi162 2024-01-15T22:35:05.435 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:35:05 smithi162 systemd[1]: Stopping Ceph mon.smithi162 for eb17a8d4-b3f4-11ee-95ad-87774f69a715... 2024-01-15T22:35:05.436 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:35:05 smithi162 bash[50152]: Error: no container with name or ID ceph-eb17a8d4-b3f4-11ee-95ad-87774f69a715-mon.smithi162 found: no such container 2024-01-15T22:35:05.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:35:05 smithi162 conmon[39755]: debug 2024-01-15T22:35:05.434+0000 7faba8c3a700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.smithi162 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-15T22:35:05.698 INFO:journalctl@ceph.mon.smithi162.smithi162.stdout:Jan 15 22:35:05 smithi162 conmon[39755]: debug 2024-01-15T22:35:05.434+0000 7faba8c3a700 -1 mon.smithi162@1(peon) e2 *** Got Signal Terminated *** 2024-01-15T22:35:06.348 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb17a8d4-b3f4-11ee-95ad-87774f69a715@mon.smithi162.service' 2024-01-15T22:35:06.394 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-15T22:35:06.394 INFO:tasks.cephadm.mon.smithi162:Stopped mon.smithi162 2024-01-15T22:35:06.394 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 --force --keep-logs 2024-01-15T22:35:11.928 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 --force --keep-logs 2024-01-15T22:35:16.589 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-15T22:35:16.626 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-15T22:35:16.657 INFO:tasks.cephadm:Archiving crash dumps... 2024-01-15T22:35:16.659 DEBUG:teuthology.misc:Transferring archived files from smithi099:/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-01-15_22:02:29-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7517238/remote/ubuntu@smithi099.front.sepia.ceph.com/crash 2024-01-15T22:35:16.659 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar c -f - -C /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash -- . 2024-01-15T22:35:16.776 INFO:teuthology.orchestra.run.smithi099.stderr:tar: /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash: Cannot open: No such file or directory 2024-01-15T22:35:16.777 INFO:teuthology.orchestra.run.smithi099.stderr:tar: Error is not recoverable: exiting now 2024-01-15T22:35:16.778 DEBUG:teuthology.misc:Transferring archived files from smithi162:/var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-01-15_22:02:29-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7517238/remote/ubuntu@smithi162.front.sepia.ceph.com/crash 2024-01-15T22:35:16.778 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash -- . 2024-01-15T22:35:16.813 INFO:teuthology.orchestra.run.smithi162.stderr:tar: /var/lib/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/crash: Cannot open: No such file or directory 2024-01-15T22:35:16.813 INFO:teuthology.orchestra.run.smithi162.stderr:tar: Error is not recoverable: exiting now 2024-01-15T22:35:16.814 INFO:tasks.cephadm:Checking cluster log for badness... 2024-01-15T22:35:16.814 DEBUG:teuthology.orchestra.run.smithi099:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-01-15T22:35:16.844 INFO:teuthology.orchestra.run.smithi099.stderr:grep: /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/ceph.log: No such file or directory 2024-01-15T22:35:16.846 INFO:tasks.cephadm:Compressing logs... 2024-01-15T22:35:16.846 DEBUG:teuthology.orchestra.run.smithi099:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-15T22:35:16.888 DEBUG:teuthology.orchestra.run.smithi162:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-15T22:35:16.915 INFO:teuthology.orchestra.run.smithi099.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-01-15T22:35:16.915 INFO:teuthology.orchestra.run.smithi099.stderr:: No such file or directory 2024-01-15T22:35:16.916 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/ceph-volume.log 2024-01-15T22:35:16.916 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/ceph-mon.smithi099.log 2024-01-15T22:35:16.916 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/ceph-mgr.smithi099.cmjeyn.log 2024-01-15T22:35:16.917 INFO:teuthology.orchestra.run.smithi162.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-01-15T22:35:16.917 INFO:teuthology.orchestra.run.smithi162.stderr:: No such file or directory 2024-01-15T22:35:16.917 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/ceph-mgr.smithi162.vbobpt.log 2024-01-15T22:35:16.917 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/ceph-volume.log 2024-01-15T22:35:16.917 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/ceph-mgr.smithi162.vbobpt.log: gzip -5 --verbose -- /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/ceph-mon.smithi162.log 2024-01-15T22:35:16.918 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/ceph-volume.log: /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/ceph-mon.smithi162.log: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-01-15T22:35:16.919 INFO:teuthology.orchestra.run.smithi162.stderr: 91.7% -- replaced with /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/ceph-mgr.smithi162.vbobpt.log.gz 2024-01-15T22:35:16.921 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/ceph-volume.log: /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/ceph-mon.smithi099.log: /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/ceph-mgr.smithi099.cmjeyn.log: 94.0% -- replaced with /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/ceph-volume.log.gz 2024-01-15T22:35:16.922 INFO:teuthology.orchestra.run.smithi162.stderr: 94.0% -- replaced with /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/ceph-volume.log.gz 2024-01-15T22:35:16.923 INFO:teuthology.orchestra.run.smithi099.stderr: 93.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-01-15T22:35:16.989 INFO:teuthology.orchestra.run.smithi162.stderr: 92.5% -- replaced with /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/ceph-mon.smithi162.log.gz 2024-01-15T22:35:16.990 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-01-15T22:35:16.991 INFO:teuthology.orchestra.run.smithi162.stderr:real 0m0.086s 2024-01-15T22:35:16.991 INFO:teuthology.orchestra.run.smithi162.stderr:user 0m0.090s 2024-01-15T22:35:16.991 INFO:teuthology.orchestra.run.smithi162.stderr:sys 0m0.017s 2024-01-15T22:35:17.006 INFO:teuthology.orchestra.run.smithi099.stderr: 91.0% -- replaced with /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/ceph-mgr.smithi099.cmjeyn.log.gz 2024-01-15T22:35:17.162 INFO:teuthology.orchestra.run.smithi099.stderr: 89.8% -- replaced with /var/log/ceph/eb17a8d4-b3f4-11ee-95ad-87774f69a715/ceph-mon.smithi099.log.gz 2024-01-15T22:35:17.163 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-01-15T22:35:17.164 INFO:teuthology.orchestra.run.smithi099.stderr:real 0m0.260s 2024-01-15T22:35:17.164 INFO:teuthology.orchestra.run.smithi099.stderr:user 0m0.344s 2024-01-15T22:35:17.164 INFO:teuthology.orchestra.run.smithi099.stderr:sys 0m0.030s 2024-01-15T22:35:17.164 INFO:tasks.cephadm:Archiving logs... 2024-01-15T22:35:17.164 DEBUG:teuthology.misc:Transferring archived files from smithi099:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-15_22:02:29-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7517238/remote/ubuntu@smithi099.front.sepia.ceph.com/log 2024-01-15T22:35:17.165 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-15T22:35:17.302 DEBUG:teuthology.misc:Transferring archived files from smithi162:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-15_22:02:29-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7517238/remote/ubuntu@smithi162.front.sepia.ceph.com/log 2024-01-15T22:35:17.303 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-15T22:35:17.347 INFO:tasks.cephadm:Removing cluster... 2024-01-15T22:35:17.347 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 --force 2024-01-15T22:35:17.784 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eb17a8d4-b3f4-11ee-95ad-87774f69a715 --force 2024-01-15T22:35:18.218 INFO:tasks.cephadm:Removing cephadm ... 2024-01-15T22:35:18.219 DEBUG:teuthology.orchestra.run.smithi099:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-15T22:35:18.238 DEBUG:teuthology.orchestra.run.smithi162:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-15T22:35:18.255 INFO:tasks.cephadm:Teardown complete 2024-01-15T22:35:18.255 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 779, in ceph_osds while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (121) after waiting for 120 seconds 2024-01-15T22:35:18.478 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=66335aae8e96449e9c8c853eca74a934 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 779, in ceph_osds while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (121) after waiting for 120 seconds 2024-01-15T22:35:18.481 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-01-15T22:35:18.492 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop 2024-01-15T22:35:18.499 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi099:/dev/vg_nvme/lv_1... 2024-01-15T22:35:18.499 DEBUG:teuthology.orchestra.run.smithi099:> sudo nvme disconnect -n lv_1 2024-01-15T22:35:18.641 INFO:teuthology.orchestra.run.smithi099.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-01-15T22:35:18.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T22:35:18.643 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi099:/dev/vg_nvme/lv_2... 2024-01-15T22:35:18.643 DEBUG:teuthology.orchestra.run.smithi099:> sudo nvme disconnect -n lv_2 2024-01-15T22:35:18.777 INFO:teuthology.orchestra.run.smithi099.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-01-15T22:35:18.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T22:35:18.778 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi099:/dev/vg_nvme/lv_3... 2024-01-15T22:35:18.778 DEBUG:teuthology.orchestra.run.smithi099:> sudo nvme disconnect -n lv_3 2024-01-15T22:35:18.894 INFO:teuthology.orchestra.run.smithi099.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-01-15T22:35:18.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T22:35:18.895 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi099:/dev/vg_nvme/lv_4... 2024-01-15T22:35:18.895 DEBUG:teuthology.orchestra.run.smithi099:> sudo nvme disconnect -n lv_4 2024-01-15T22:35:19.008 INFO:teuthology.orchestra.run.smithi099.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-01-15T22:35:19.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T22:35:19.011 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-01-15T22:35:19.011 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/scratch_devs 2024-01-15T22:35:19.037 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi162:/dev/vg_nvme/lv_1... 2024-01-15T22:35:19.037 DEBUG:teuthology.orchestra.run.smithi162:> sudo nvme disconnect -n lv_1 2024-01-15T22:35:19.170 INFO:teuthology.orchestra.run.smithi162.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-01-15T22:35:19.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T22:35:19.172 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi162:/dev/vg_nvme/lv_2... 2024-01-15T22:35:19.172 DEBUG:teuthology.orchestra.run.smithi162:> sudo nvme disconnect -n lv_2 2024-01-15T22:35:19.287 INFO:teuthology.orchestra.run.smithi162.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-01-15T22:35:19.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T22:35:19.289 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi162:/dev/vg_nvme/lv_3... 2024-01-15T22:35:19.289 DEBUG:teuthology.orchestra.run.smithi162:> sudo nvme disconnect -n lv_3 2024-01-15T22:35:19.426 INFO:teuthology.orchestra.run.smithi162.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-01-15T22:35:19.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T22:35:19.428 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi162:/dev/vg_nvme/lv_4... 2024-01-15T22:35:19.428 DEBUG:teuthology.orchestra.run.smithi162:> sudo nvme disconnect -n lv_4 2024-01-15T22:35:19.547 INFO:teuthology.orchestra.run.smithi162.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-01-15T22:35:19.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T22:35:19.549 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-01-15T22:35:19.549 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/scratch_devs 2024-01-15T22:35:19.578 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-15T22:35:19.594 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-15T22:35:19.594 DEBUG:teuthology.orchestra.run.smithi099:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-15T22:35:19.597 DEBUG:teuthology.orchestra.run.smithi162:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-15T22:35:19.612 INFO:teuthology.orchestra.run.smithi099.stderr:bash: ntpq: command not found 2024-01-15T22:35:19.617 INFO:teuthology.orchestra.run.smithi099.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-15T22:35:19.617 INFO:teuthology.orchestra.run.smithi099.stdout:=============================================================================== 2024-01-15T22:35:19.617 INFO:teuthology.orchestra.run.smithi099.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 58 -935us[ -935us] +/- 69ms 2024-01-15T22:35:19.618 INFO:teuthology.orchestra.run.smithi099.stdout:^* hv02.front.sepia.ceph.com 3 6 377 58 -15us[ -35us] +/- 51ms 2024-01-15T22:35:19.618 INFO:teuthology.orchestra.run.smithi099.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 57 +1279us[+1279us] +/- 88ms 2024-01-15T22:35:19.618 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-15T22:35:19.635 INFO:teuthology.orchestra.run.smithi162.stderr:bash: ntpq: command not found 2024-01-15T22:35:19.639 INFO:teuthology.orchestra.run.smithi162.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-15T22:35:19.639 INFO:teuthology.orchestra.run.smithi162.stdout:=============================================================================== 2024-01-15T22:35:19.639 INFO:teuthology.orchestra.run.smithi162.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 56 -938us[ -938us] +/- 69ms 2024-01-15T22:35:19.639 INFO:teuthology.orchestra.run.smithi162.stdout:^* hv02.front.sepia.ceph.com 3 6 377 58 -23us[ -35us] +/- 51ms 2024-01-15T22:35:19.640 INFO:teuthology.orchestra.run.smithi162.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 60 +1284us[+1271us] +/- 88ms 2024-01-15T22:35:19.640 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-15T22:35:19.642 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-15T22:35:19.657 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-15T22:35:19.659 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-15T22:35:19.676 DEBUG:teuthology.orchestra.run.smithi099:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-15T22:35:19.742 DEBUG:teuthology.orchestra.run.smithi162:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-15T22:35:19.809 DEBUG:teuthology.orchestra.run.smithi099:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-15T22:35:19.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T22:35:19.839 DEBUG:teuthology.orchestra.run.smithi162:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-15T22:35:19.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T22:35:19.871 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-15T22:35:19.881 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-15T22:35:19.895 INFO:teuthology.task.internal:Duration was 1258.684110 seconds 2024-01-15T22:35:19.896 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-15T22:35:19.912 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-15T22:35:19.913 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-15T22:35:19.915 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-15T22:35:19.959 INFO:teuthology.orchestra.run.smithi099.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-15T22:35:19.964 INFO:teuthology.orchestra.run.smithi162.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-15T22:35:20.211 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-15T22:35:20.211 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi099.front.sepia.ceph.com 2024-01-15T22:35:20.212 DEBUG:teuthology.orchestra.run.smithi099:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-15T22:35:20.233 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi162.front.sepia.ceph.com 2024-01-15T22:35:20.234 DEBUG:teuthology.orchestra.run.smithi162:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-15T22:35:20.256 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-15T22:35:20.257 DEBUG:teuthology.orchestra.run.smithi099:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-15T22:35:20.276 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-15T22:35:20.351 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-15T22:35:20.352 DEBUG:teuthology.orchestra.run.smithi099:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-15T22:35:20.354 DEBUG:teuthology.orchestra.run.smithi162:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-15T22:35:20.500 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-15T22:35:20.511 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-15T22:35:20.511 DEBUG:teuthology.orchestra.run.smithi099:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-15T22:35:20.539 DEBUG:teuthology.orchestra.run.smithi162:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-15T22:35:20.566 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-15T22:35:20.579 DEBUG:teuthology.orchestra.run.smithi099:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-15T22:35:20.582 DEBUG:teuthology.orchestra.run.smithi162:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-15T22:35:20.606 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern = core 2024-01-15T22:35:20.633 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = core 2024-01-15T22:35:20.657 DEBUG:teuthology.orchestra.run.smithi099:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-15T22:35:20.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T22:35:20.711 DEBUG:teuthology.orchestra.run.smithi162:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-15T22:35:20.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T22:35:20.727 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-15T22:35:20.740 INFO:teuthology.task.internal:Transferring archived files... 2024-01-15T22:35:20.741 DEBUG:teuthology.misc:Transferring archived files from smithi099:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-15_22:02:29-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7517238/remote/smithi099 2024-01-15T22:35:20.741 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-15T22:35:20.798 DEBUG:teuthology.misc:Transferring archived files from smithi162:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-15_22:02:29-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7517238/remote/smithi162 2024-01-15T22:35:20.799 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-15T22:35:20.845 INFO:teuthology.task.internal:Removing archive directory... 2024-01-15T22:35:20.846 DEBUG:teuthology.orchestra.run.smithi099:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-15T22:35:20.848 DEBUG:teuthology.orchestra.run.smithi162:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-15T22:35:20.895 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-15T22:35:20.910 INFO:teuthology.task.internal:Not uploading archives. 2024-01-15T22:35:20.910 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-15T22:35:20.927 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-15T22:35:20.928 DEBUG:teuthology.orchestra.run.smithi099:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-15T22:35:20.930 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-15T22:35:20.944 INFO:teuthology.orchestra.run.smithi099.stdout: 262179 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 15 22:35 /home/ubuntu/cephtest 2024-01-15T22:35:20.951 INFO:teuthology.orchestra.run.smithi162.stdout: 262157 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 15 22:35 /home/ubuntu/cephtest 2024-01-15T22:35:20.953 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-15T22:35:20.966 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-15T22:35:20.996 INFO:teuthology.nuke:Checking targets against current locks 2024-01-15T22:35:21.023 DEBUG:teuthology.nuke:shortname: smithi099 2024-01-15T22:35:21.024 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-15T22:35:21.047 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi099.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-15_22:02:29-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7517238', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-15 22:04:58.502999', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMot5Qaeq01nq2YXbQR2TDuQS4ijbfr9CnnRbMpJBClwb7TSaR9s+biN/qOmzGjqSKwc0rJptpRmHgw32mRGdUE='} 2024-01-15T22:35:21.051 DEBUG:teuthology.nuke:shortname: smithi162 2024-01-15T22:35:21.051 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-15T22:35:21.079 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi162.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-15_22:02:29-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7517238', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-15 22:04:58.501950', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE9yp1RriOwRgcLJDD6X2eivuvf40mVddiFtyoo0tvH3VNXb4FgLO/0b5OsnIpZcsKUUvRfKEaq6ONa3eaRZJIc='} 2024-01-15T22:35:21.107 INFO:teuthology.orchestra.console.smithi099:Power off 2024-01-15T22:35:21.107 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-15T22:35:21.132 DEBUG:teuthology.orchestra.console.smithi099:power off output: Chassis Power Control: Down/Off 2024-01-15T22:35:21.132 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-15T22:35:21.154 INFO:teuthology.orchestra.console.smithi162:Power off 2024-01-15T22:35:21.154 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-15T22:35:21.168 DEBUG:teuthology.orchestra.console.smithi099:check power output: Chassis Power is on 2024-01-15T22:35:21.183 DEBUG:teuthology.orchestra.console.smithi162:power off output: Chassis Power Control: Down/Off 2024-01-15T22:35:21.184 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-15T22:35:21.208 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is on 2024-01-15T22:35:25.169 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-15T22:35:25.210 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-15T22:35:25.280 DEBUG:teuthology.orchestra.console.smithi099:check power output: Chassis Power is on 2024-01-15T22:35:25.320 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is on 2024-01-15T22:35:29.284 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-15T22:35:29.325 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-15T22:35:29.397 DEBUG:teuthology.orchestra.console.smithi099:check power output: Chassis Power is off 2024-01-15T22:35:29.437 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is off 2024-01-15T22:35:29.498 INFO:teuthology.orchestra.console.smithi099:Power off completed 2024-01-15T22:35:29.538 INFO:teuthology.orchestra.console.smithi162:Power off completed 2024-01-15T22:35:29.642 INFO:teuthology.run:Summary data: description: rados/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/basic 3-final} duration: 1258.6841099262238 failure_reason: reached maximum tries (121) after waiting for 120 seconds owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=66335aae8e96449e9c8c853eca74a934 status: fail success: false 2024-01-15T22:35:29.643 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-15T22:35:29.723 INFO:teuthology.run:FAIL