2024-01-13T08:47:10.987 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-13T08:47:10.987 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515791 --name yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi --owner scheduled_yuriw@teuthology --verbose --description rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/repave-all} -- /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515791/orig.config.yaml 2024-01-13T08:47:11.014 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-13T08:47:11.091 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515791 branch: wip-yuri2-testing-2024-01-12-1128-pacific description: rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/repave-all} email: yweinste@redhat.com first_in_suite: false job_id: '7515791' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-yuri2-testing-2024-01-12-1128-pacific ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 445562ab4bc3ddfb386936119050695810860bcb ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 445562ab4bc3ddfb386936119050695810860bcb selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri2-testing-2024-01-12-1128-pacific sha1: 445562ab4bc3ddfb386936119050695810860bcb owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - - host.b - client.1 seed: 5246 sha1: 445562ab4bc3ddfb386936119050695810860bcb sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri2-testing-2024-01-12-1128-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 445562ab4bc3ddfb386936119050695810860bcb targets: smithi077.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN8NLaKSd0BQrCfI69gFZWN0W/k4ulGQUWWPxeP9PPBdluMvnUoDRb0h4DX/BuL19Qag4kCSgSg5VrT/vRA2deg= smithi141.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDAeiwsfWiWyk0KzxhIy3Ic4FDNhZ1DIHGf/d6PpuegFVnBW9OVMS6QlEN4luyqgl/19FSjxP7hQsEQa4BneOyU= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - nvme_loop: null - cephadm: roleless: true - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch ls | grep '^osd.all-available-devices ' - cephadm.shell: host.a: - "set -e\nset -x\nceph orch ps\nceph orch device ls\nceph osd tree\nfor osd in\ \ `ceph osd ls` ; do\n ceph orch osd rm $osd --force --zap --replace\ndone\n\ while ceph orch osd rm ls | wc | grep ^1 ; do sleep 10 ; done\n" teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-12_21:52:25 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-01-13T08:47:11.091 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa; will attempt to use it 2024-01-13T08:47:11.092 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks 2024-01-13T08:47:11.092 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-13T08:47:11.094 INFO:teuthology.task.internal:Checking packages... 2024-01-13T08:47:11.117 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '445562ab4bc3ddfb386936119050695810860bcb' 2024-01-13T08:47:11.118 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-13T08:47:11.118 INFO:teuthology.packaging:ref: None 2024-01-13T08:47:11.118 INFO:teuthology.packaging:tag: None 2024-01-13T08:47:11.118 INFO:teuthology.packaging:branch: wip-yuri2-testing-2024-01-12-1128-pacific 2024-01-13T08:47:11.118 INFO:teuthology.packaging:sha1: 445562ab4bc3ddfb386936119050695810860bcb 2024-01-13T08:47:11.118 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri2-testing-2024-01-12-1128-pacific 2024-01-13T08:47:11.497 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-456.g445562ab 2024-01-13T08:47:11.498 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-13T08:47:11.505 INFO:teuthology.task.internal:no buildpackages task found 2024-01-13T08:47:11.505 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-13T08:47:11.526 INFO:teuthology.task.internal:Saving configuration 2024-01-13T08:47:11.537 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-13T08:47:11.562 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-13T08:47:11.589 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515791', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-13 08:42:03.140021', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN8NLaKSd0BQrCfI69gFZWN0W/k4ulGQUWWPxeP9PPBdluMvnUoDRb0h4DX/BuL19Qag4kCSgSg5VrT/vRA2deg='} 2024-01-13T08:47:11.614 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi141.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515791', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-13 08:42:03.142064', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDAeiwsfWiWyk0KzxhIy3Ic4FDNhZ1DIHGf/d6PpuegFVnBW9OVMS6QlEN4luyqgl/19FSjxP7hQsEQa4BneOyU='} 2024-01-13T08:47:11.614 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-13T08:47:11.620 INFO:teuthology.task.internal:roles: ubuntu@smithi077.front.sepia.ceph.com - ['host.a', 'client.0'] 2024-01-13T08:47:11.620 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.b', 'client.1'] 2024-01-13T08:47:11.620 INFO:teuthology.run_tasks:Running task console_log... 2024-01-13T08:47:11.690 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f929272dee0>, signals=[15]) 2024-01-13T08:47:11.691 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-13T08:47:11.696 INFO:teuthology.task.internal:Opening connections... 2024-01-13T08:47:11.697 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi077.front.sepia.ceph.com 2024-01-13T08:47:11.698 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-13T08:47:11.775 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi141.front.sepia.ceph.com 2024-01-13T08:47:11.775 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi141.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-13T08:47:11.862 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-13T08:47:11.869 DEBUG:teuthology.orchestra.run.smithi077:> uname -m 2024-01-13T08:47:11.889 INFO:teuthology.orchestra.run.smithi077.stdout:x86_64 2024-01-13T08:47:11.889 DEBUG:teuthology.orchestra.run.smithi077:> cat /etc/os-release 2024-01-13T08:47:11.950 INFO:teuthology.orchestra.run.smithi077.stdout:NAME="Red Hat Enterprise Linux" 2024-01-13T08:47:11.950 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION="8.6 (Ootpa)" 2024-01-13T08:47:11.950 INFO:teuthology.orchestra.run.smithi077.stdout:ID="rhel" 2024-01-13T08:47:11.950 INFO:teuthology.orchestra.run.smithi077.stdout:ID_LIKE="fedora" 2024-01-13T08:47:11.951 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION_ID="8.6" 2024-01-13T08:47:11.951 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM_ID="platform:el8" 2024-01-13T08:47:11.951 INFO:teuthology.orchestra.run.smithi077.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-01-13T08:47:11.951 INFO:teuthology.orchestra.run.smithi077.stdout:ANSI_COLOR="0;31" 2024-01-13T08:47:11.951 INFO:teuthology.orchestra.run.smithi077.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-01-13T08:47:11.951 INFO:teuthology.orchestra.run.smithi077.stdout:HOME_URL="https://www.redhat.com/" 2024-01-13T08:47:11.951 INFO:teuthology.orchestra.run.smithi077.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-01-13T08:47:11.951 INFO:teuthology.orchestra.run.smithi077.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-13T08:47:11.951 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T08:47:11.951 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-13T08:47:11.951 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-01-13T08:47:11.951 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-01-13T08:47:11.952 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-01-13T08:47:11.953 INFO:teuthology.lock.ops:Updating smithi077.front.sepia.ceph.com on lock server 2024-01-13T08:47:11.972 DEBUG:teuthology.orchestra.run.smithi141:> uname -m 2024-01-13T08:47:11.992 INFO:teuthology.orchestra.run.smithi141.stdout:x86_64 2024-01-13T08:47:11.993 DEBUG:teuthology.orchestra.run.smithi141:> cat /etc/os-release 2024-01-13T08:47:12.053 INFO:teuthology.orchestra.run.smithi141.stdout:NAME="Red Hat Enterprise Linux" 2024-01-13T08:47:12.053 INFO:teuthology.orchestra.run.smithi141.stdout:VERSION="8.6 (Ootpa)" 2024-01-13T08:47:12.053 INFO:teuthology.orchestra.run.smithi141.stdout:ID="rhel" 2024-01-13T08:47:12.053 INFO:teuthology.orchestra.run.smithi141.stdout:ID_LIKE="fedora" 2024-01-13T08:47:12.054 INFO:teuthology.orchestra.run.smithi141.stdout:VERSION_ID="8.6" 2024-01-13T08:47:12.054 INFO:teuthology.orchestra.run.smithi141.stdout:PLATFORM_ID="platform:el8" 2024-01-13T08:47:12.054 INFO:teuthology.orchestra.run.smithi141.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-01-13T08:47:12.054 INFO:teuthology.orchestra.run.smithi141.stdout:ANSI_COLOR="0;31" 2024-01-13T08:47:12.054 INFO:teuthology.orchestra.run.smithi141.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-01-13T08:47:12.054 INFO:teuthology.orchestra.run.smithi141.stdout:HOME_URL="https://www.redhat.com/" 2024-01-13T08:47:12.055 INFO:teuthology.orchestra.run.smithi141.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-01-13T08:47:12.055 INFO:teuthology.orchestra.run.smithi141.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-13T08:47:12.055 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-13T08:47:12.055 INFO:teuthology.orchestra.run.smithi141.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-13T08:47:12.055 INFO:teuthology.orchestra.run.smithi141.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-01-13T08:47:12.055 INFO:teuthology.orchestra.run.smithi141.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-01-13T08:47:12.056 INFO:teuthology.orchestra.run.smithi141.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-01-13T08:47:12.056 INFO:teuthology.lock.ops:Updating smithi141.front.sepia.ceph.com on lock server 2024-01-13T08:47:12.081 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-13T08:47:12.089 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-13T08:47:12.096 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-13T08:47:12.096 DEBUG:teuthology.orchestra.run.smithi077:> test '!' -e /home/ubuntu/cephtest 2024-01-13T08:47:12.098 DEBUG:teuthology.orchestra.run.smithi141:> test '!' -e /home/ubuntu/cephtest 2024-01-13T08:47:12.117 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-13T08:47:12.122 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-13T08:47:12.122 DEBUG:teuthology.orchestra.run.smithi077:> test -z $(ls -A /var/lib/ceph) 2024-01-13T08:47:12.159 DEBUG:teuthology.orchestra.run.smithi141:> test -z $(ls -A /var/lib/ceph) 2024-01-13T08:47:12.203 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-13T08:47:12.261 INFO:teuthology.run_tasks:Running task kernel... 2024-01-13T08:47:12.275 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-01-13T08:47:12.275 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-01-13T08:47:12.275 DEBUG:teuthology.orchestra.run.smithi077:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-13T08:47:12.276 DEBUG:teuthology.orchestra.run.smithi141:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-13T08:47:12.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T08:47:12.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T08:47:12.295 DEBUG:teuthology.orchestra.run.smithi077:> uname -r 2024-01-13T08:47:12.295 DEBUG:teuthology.orchestra.run.smithi141:> uname -r 2024-01-13T08:47:12.354 INFO:teuthology.orchestra.run.smithi077.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-01-13T08:47:12.354 INFO:teuthology.task.kernel:Running kernel on smithi077: 4.18.0-513.9.1.el8_9.x86_64 2024-01-13T08:47:12.354 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum install -y kernel 2024-01-13T08:47:12.357 INFO:teuthology.orchestra.run.smithi141.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-01-13T08:47:12.357 INFO:teuthology.task.kernel:Running kernel on smithi141: 4.18.0-513.9.1.el8_9.x86_64 2024-01-13T08:47:12.357 DEBUG:teuthology.orchestra.run.smithi141:> sudo yum install -y kernel 2024-01-13T08:47:13.836 INFO:teuthology.orchestra.run.smithi141.stdout:Updating Subscription Management repositories. 2024-01-13T08:47:13.836 INFO:teuthology.orchestra.run.smithi141.stdout:Unable to read consumer identity 2024-01-13T08:47:13.871 INFO:teuthology.orchestra.run.smithi077.stdout:Updating Subscription Management repositories. 2024-01-13T08:47:13.871 INFO:teuthology.orchestra.run.smithi077.stdout:Unable to read consumer identity 2024-01-13T08:47:14.135 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T08:47:14.136 INFO:teuthology.orchestra.run.smithi077.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-13T08:47:14.136 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T08:47:14.160 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-13T08:47:14.160 INFO:teuthology.orchestra.run.smithi141.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-13T08:47:14.160 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-13T08:47:16.092 INFO:teuthology.orchestra.run.smithi141.stdout:Copr repo for python3-asyncssh owned by ceph 8.5 kB/s | 3.5 kB 00:00 2024-01-13T08:47:16.114 INFO:teuthology.orchestra.run.smithi077.stdout:Copr repo for python3-asyncssh owned by ceph 8.2 kB/s | 3.5 kB 00:00 2024-01-13T08:47:18.617 INFO:teuthology.orchestra.run.smithi141.stdout:Extra Packages for Enterprise Linux 6.5 MB/s | 16 MB 00:02 2024-01-13T08:47:19.892 INFO:teuthology.orchestra.run.smithi077.stdout:Extra Packages for Enterprise Linux 4.3 MB/s | 16 MB 00:03 2024-01-13T08:47:22.167 INFO:teuthology.orchestra.run.smithi141.stdout:lab-extras 356 kB/s | 24 kB 00:00 2024-01-13T08:47:22.911 INFO:teuthology.orchestra.run.smithi141.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-01-13T08:47:22.912 INFO:teuthology.orchestra.run.smithi141.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-01-13T08:47:22.989 INFO:teuthology.orchestra.run.smithi141.stdout:Dependencies resolved. 2024-01-13T08:47:22.990 INFO:teuthology.orchestra.run.smithi141.stdout:Nothing to do. 2024-01-13T08:47:22.990 INFO:teuthology.orchestra.run.smithi141.stdout:Complete! 2024-01-13T08:47:23.139 DEBUG:teuthology.orchestra.run.smithi141:> echo no | sudo yum reinstall kernel || true 2024-01-13T08:47:23.475 INFO:teuthology.orchestra.run.smithi141.stdout:Updating Subscription Management repositories. 2024-01-13T08:47:23.475 INFO:teuthology.orchestra.run.smithi141.stdout:Unable to read consumer identity 2024-01-13T08:47:23.523 INFO:teuthology.orchestra.run.smithi077.stdout:lab-extras 296 kB/s | 24 kB 00:00 2024-01-13T08:47:23.700 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-13T08:47:23.700 INFO:teuthology.orchestra.run.smithi141.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-13T08:47:23.700 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-13T08:47:24.019 INFO:teuthology.orchestra.run.smithi141.stdout:Last metadata expiration check: 0:00:02 ago on Sat 13 Jan 2024 08:47:22 AM UTC. 2024-01-13T08:47:24.097 INFO:teuthology.orchestra.run.smithi141.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-01-13T08:47:24.109 INFO:teuthology.orchestra.run.smithi141.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-01-13T08:47:24.149 INFO:teuthology.orchestra.run.smithi141.stderr:Error: No packages marked for reinstall. 2024-01-13T08:47:24.196 DEBUG:teuthology.orchestra.run.smithi141:> sudo yum reinstall -y kernel || true 2024-01-13T08:47:24.275 INFO:teuthology.orchestra.run.smithi077.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-01-13T08:47:24.275 INFO:teuthology.orchestra.run.smithi077.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-01-13T08:47:24.358 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-01-13T08:47:24.359 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2024-01-13T08:47:24.359 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-01-13T08:47:24.487 DEBUG:teuthology.orchestra.run.smithi077:> echo no | sudo yum reinstall kernel || true 2024-01-13T08:47:24.532 INFO:teuthology.orchestra.run.smithi141.stdout:Updating Subscription Management repositories. 2024-01-13T08:47:24.532 INFO:teuthology.orchestra.run.smithi141.stdout:Unable to read consumer identity 2024-01-13T08:47:24.764 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-13T08:47:24.764 INFO:teuthology.orchestra.run.smithi141.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-13T08:47:24.764 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-13T08:47:24.826 INFO:teuthology.orchestra.run.smithi077.stdout:Updating Subscription Management repositories. 2024-01-13T08:47:24.826 INFO:teuthology.orchestra.run.smithi077.stdout:Unable to read consumer identity 2024-01-13T08:47:25.006 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T08:47:25.006 INFO:teuthology.orchestra.run.smithi077.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-13T08:47:25.006 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T08:47:25.070 INFO:teuthology.orchestra.run.smithi141.stdout:Last metadata expiration check: 0:00:03 ago on Sat 13 Jan 2024 08:47:22 AM UTC. 2024-01-13T08:47:25.134 INFO:teuthology.orchestra.run.smithi141.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-01-13T08:47:25.134 INFO:teuthology.orchestra.run.smithi141.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-01-13T08:47:25.172 INFO:teuthology.orchestra.run.smithi141.stderr:Error: No packages marked for reinstall. 2024-01-13T08:47:25.224 DEBUG:teuthology.orchestra.run.smithi141:> rpm -q kernel | sort -rV | head -n 1 2024-01-13T08:47:25.314 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:00:02 ago on Sat 13 Jan 2024 08:47:23 AM UTC. 2024-01-13T08:47:25.393 INFO:teuthology.orchestra.run.smithi141.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-01-13T08:47:25.394 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-01-13T08:47:25.394 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-01-13T08:47:25.394 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-01-13T08:47:25.394 DEBUG:teuthology.orchestra.run.smithi141:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-13T08:47:25.398 INFO:teuthology.orchestra.run.smithi077.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-01-13T08:47:25.413 INFO:teuthology.orchestra.run.smithi077.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-01-13T08:47:25.437 INFO:teuthology.orchestra.run.smithi141.stdout:ttyS1 2024-01-13T08:47:25.450 DEBUG:teuthology.parallel:result is None 2024-01-13T08:47:25.451 INFO:teuthology.orchestra.run.smithi077.stderr:Error: No packages marked for reinstall. 2024-01-13T08:47:25.497 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum reinstall -y kernel || true 2024-01-13T08:47:25.833 INFO:teuthology.orchestra.run.smithi077.stdout:Updating Subscription Management repositories. 2024-01-13T08:47:25.833 INFO:teuthology.orchestra.run.smithi077.stdout:Unable to read consumer identity 2024-01-13T08:47:26.094 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T08:47:26.094 INFO:teuthology.orchestra.run.smithi077.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-13T08:47:26.094 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T08:47:26.394 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:00:03 ago on Sat 13 Jan 2024 08:47:23 AM UTC. 2024-01-13T08:47:26.456 INFO:teuthology.orchestra.run.smithi077.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-01-13T08:47:26.457 INFO:teuthology.orchestra.run.smithi077.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-01-13T08:47:26.491 INFO:teuthology.orchestra.run.smithi077.stderr:Error: No packages marked for reinstall. 2024-01-13T08:47:26.539 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q kernel | sort -rV | head -n 1 2024-01-13T08:47:26.707 INFO:teuthology.orchestra.run.smithi077.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-01-13T08:47:26.707 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-01-13T08:47:26.707 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-01-13T08:47:26.707 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-01-13T08:47:26.707 DEBUG:teuthology.orchestra.run.smithi077:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-13T08:47:26.739 INFO:teuthology.orchestra.run.smithi077.stdout:ttyS1 2024-01-13T08:47:26.754 DEBUG:teuthology.parallel:result is None 2024-01-13T08:47:26.754 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-13T08:47:26.761 INFO:teuthology.task.internal:Creating test directory... 2024-01-13T08:47:26.761 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-13T08:47:26.797 DEBUG:teuthology.orchestra.run.smithi141:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-13T08:47:26.817 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-13T08:47:26.824 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-13T08:47:26.834 INFO:teuthology.task.internal:Creating archive directory... 2024-01-13T08:47:26.834 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-13T08:47:26.853 DEBUG:teuthology.orchestra.run.smithi141:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-13T08:47:26.898 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-13T08:47:26.907 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-13T08:47:26.907 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-13T08:47:26.916 DEBUG:teuthology.orchestra.run.smithi141:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-13T08:47:26.947 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-13T08:47:26.958 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-13T08:47:26.973 INFO:teuthology.orchestra.run.smithi141.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-13T08:47:26.985 INFO:teuthology.orchestra.run.smithi141.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-13T08:47:26.986 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-13T08:47:26.995 INFO:teuthology.task.internal:Configuring sudo... 2024-01-13T08:47:26.995 DEBUG:teuthology.orchestra.run.smithi077:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-13T08:47:27.001 DEBUG:teuthology.orchestra.run.smithi141:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-13T08:47:27.055 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-13T08:47:27.065 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-13T08:47:27.066 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-13T08:47:27.067 DEBUG:teuthology.orchestra.run.smithi141:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-13T08:47:27.113 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-13T08:47:27.163 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-13T08:47:27.216 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-13T08:47:27.300 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-13T08:47:27.342 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-13T08:47:27.342 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-13T08:47:27.408 DEBUG:teuthology.orchestra.run.smithi141:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-13T08:47:27.453 DEBUG:teuthology.orchestra.run.smithi141:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-13T08:47:27.511 DEBUG:teuthology.orchestra.run.smithi141:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-13T08:47:27.596 DEBUG:teuthology.orchestra.run.smithi141:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-13T08:47:27.639 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-13T08:47:27.639 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-13T08:47:27.707 DEBUG:teuthology.orchestra.run.smithi077:> sudo service rsyslog restart 2024-01-13T08:47:27.710 DEBUG:teuthology.orchestra.run.smithi141:> sudo service rsyslog restart 2024-01-13T08:47:27.754 INFO:teuthology.orchestra.run.smithi077.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-13T08:47:27.794 INFO:teuthology.orchestra.run.smithi141.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-13T08:47:28.143 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-13T08:47:28.153 INFO:teuthology.task.internal:Starting timer... 2024-01-13T08:47:28.153 INFO:teuthology.run_tasks:Running task pcp... 2024-01-13T08:47:28.170 INFO:teuthology.run_tasks:Running task selinux... 2024-01-13T08:47:28.178 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-01-13T08:47:28.178 DEBUG:teuthology.orchestra.run.smithi077:> sudo service auditd rotate 2024-01-13T08:47:28.243 INFO:teuthology.orchestra.run.smithi077.stdout:Rotating logs: 2024-01-13T08:47:28.245 DEBUG:teuthology.orchestra.run.smithi141:> sudo service auditd rotate 2024-01-13T08:47:28.323 INFO:teuthology.orchestra.run.smithi141.stdout:Rotating logs: 2024-01-13T08:47:28.325 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-13T08:47:28.325 DEBUG:teuthology.orchestra.run.smithi077:> /usr/sbin/getenforce 2024-01-13T08:47:28.365 INFO:teuthology.orchestra.run.smithi077.stdout:Permissive 2024-01-13T08:47:28.365 DEBUG:teuthology.orchestra.run.smithi141:> /usr/sbin/getenforce 2024-01-13T08:47:28.391 INFO:teuthology.orchestra.run.smithi141.stdout:Permissive 2024-01-13T08:47:28.391 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi077.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi141.front.sepia.ceph.com': 'permissive'} 2024-01-13T08:47:28.391 DEBUG:teuthology.orchestra.run.smithi077:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-13T08:47:28.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T08:47:28.433 DEBUG:teuthology.orchestra.run.smithi141:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-13T08:47:28.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T08:47:28.460 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-13T08:47:28.460 DEBUG:teuthology.orchestra.run.smithi077:> sudo /usr/sbin/setenforce permissive 2024-01-13T08:47:28.500 DEBUG:teuthology.orchestra.run.smithi141:> sudo /usr/sbin/setenforce permissive 2024-01-13T08:47:28.528 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-13T08:47:28.538 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-01-13T08:47:28.543 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-01-13T08:47:28.667 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-13T08:47:28.682 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-13T08:47:28.683 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi077.front.sepia.ceph.com,smithi141.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-13T08:55:21.005 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi077.front.sepia.ceph.com'), Remote(name='ubuntu@smithi141.front.sepia.ceph.com')] 2024-01-13T08:55:21.007 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-13T08:55:21.008 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-13T08:55:21.084 DEBUG:teuthology.orchestra.run.smithi077:> true 2024-01-13T08:55:21.157 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-01-13T08:55:21.157 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2024-01-13T08:55:21.158 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi141.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-13T08:55:21.234 DEBUG:teuthology.orchestra.run.smithi141:> true 2024-01-13T08:55:21.302 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi141.front.sepia.ceph.com' 2024-01-13T08:55:21.302 INFO:teuthology.run_tasks:Running task clock... 2024-01-13T08:55:21.314 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-13T08:55:21.314 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-13T08:55:21.315 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-13T08:55:21.318 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-13T08:55:21.318 DEBUG:teuthology.orchestra.run.smithi141:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-13T08:55:21.352 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-13T08:55:21.368 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-13T08:55:21.374 INFO:teuthology.orchestra.run.smithi141.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-13T08:55:21.391 INFO:teuthology.orchestra.run.smithi141.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-13T08:55:21.439 INFO:teuthology.orchestra.run.smithi077.stderr:sudo: ntpd: command not found 2024-01-13T08:55:21.452 INFO:teuthology.orchestra.run.smithi077.stdout:506 Cannot talk to daemon 2024-01-13T08:55:21.462 INFO:teuthology.orchestra.run.smithi141.stderr:sudo: ntpd: command not found 2024-01-13T08:55:21.467 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-13T08:55:21.474 INFO:teuthology.orchestra.run.smithi141.stdout:506 Cannot talk to daemon 2024-01-13T08:55:21.481 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-13T08:55:21.490 INFO:teuthology.orchestra.run.smithi141.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-13T08:55:21.504 INFO:teuthology.orchestra.run.smithi141.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-13T08:55:21.557 INFO:teuthology.orchestra.run.smithi077.stderr:bash: ntpq: command not found 2024-01-13T08:55:21.560 INFO:teuthology.orchestra.run.smithi077.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-13T08:55:21.560 INFO:teuthology.orchestra.run.smithi077.stdout:=============================================================================== 2024-01-13T08:55:21.561 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T08:55:21.561 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T08:55:21.561 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T08:55:21.561 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T08:55:21.573 INFO:teuthology.orchestra.run.smithi141.stderr:bash: ntpq: command not found 2024-01-13T08:55:21.577 INFO:teuthology.orchestra.run.smithi141.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-13T08:55:21.577 INFO:teuthology.orchestra.run.smithi141.stdout:=============================================================================== 2024-01-13T08:55:21.577 INFO:teuthology.orchestra.run.smithi141.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T08:55:21.577 INFO:teuthology.orchestra.run.smithi141.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T08:55:21.577 INFO:teuthology.orchestra.run.smithi141.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T08:55:21.577 INFO:teuthology.orchestra.run.smithi141.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T08:55:21.578 INFO:teuthology.run_tasks:Running task pexec... 2024-01-13T08:55:21.587 INFO:teuthology.task.pexec:Executing custom commands... 2024-01-13T08:55:21.587 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi077.front.sepia.ceph.com 2024-01-13T08:55:21.587 DEBUG:teuthology.orchestra.run.smithi077:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-13T08:55:21.588 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi141.front.sepia.ceph.com 2024-01-13T08:55:21.588 DEBUG:teuthology.orchestra.run.smithi141:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-13T08:55:21.978 INFO:teuthology.orchestra.run.smithi077.stdout:Updating Subscription Management repositories. 2024-01-13T08:55:21.997 INFO:teuthology.orchestra.run.smithi141.stdout:Updating Subscription Management repositories. 2024-01-13T08:55:23.888 INFO:teuthology.orchestra.run.smithi077.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 27 kB/s | 4.1 kB 00:00 2024-01-13T08:55:23.892 INFO:teuthology.orchestra.run.smithi141.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 25 kB/s | 4.1 kB 00:00 2024-01-13T08:55:24.242 INFO:teuthology.orchestra.run.smithi141.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 50 kB/s | 4.5 kB 00:00 2024-01-13T08:55:24.253 INFO:teuthology.orchestra.run.smithi077.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 29 kB/s | 4.5 kB 00:00 2024-01-13T08:55:27.198 INFO:teuthology.orchestra.run.smithi141.stdout:Dependencies resolved. 2024-01-13T08:55:27.199 INFO:teuthology.orchestra.run.smithi141.stdout:================================================================================ 2024-01-13T08:55:27.199 INFO:teuthology.orchestra.run.smithi141.stdout: Package Architecture Version Repository Size 2024-01-13T08:55:27.199 INFO:teuthology.orchestra.run.smithi141.stdout:================================================================================ 2024-01-13T08:55:27.199 INFO:teuthology.orchestra.run.smithi141.stdout:Resetting modules: 2024-01-13T08:55:27.200 INFO:teuthology.orchestra.run.smithi141.stdout: container-tools 2024-01-13T08:55:27.200 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-13T08:55:27.200 INFO:teuthology.orchestra.run.smithi141.stdout:Transaction Summary 2024-01-13T08:55:27.200 INFO:teuthology.orchestra.run.smithi141.stdout:================================================================================ 2024-01-13T08:55:27.200 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-13T08:55:27.287 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-01-13T08:55:27.288 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-01-13T08:55:27.288 INFO:teuthology.orchestra.run.smithi077.stdout: Package Architecture Version Repository Size 2024-01-13T08:55:27.288 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-01-13T08:55:27.288 INFO:teuthology.orchestra.run.smithi077.stdout:Resetting modules: 2024-01-13T08:55:27.288 INFO:teuthology.orchestra.run.smithi077.stdout: container-tools 2024-01-13T08:55:27.288 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T08:55:27.289 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-01-13T08:55:27.289 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-01-13T08:55:27.289 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T08:55:41.790 INFO:teuthology.orchestra.run.smithi141.stdout:Complete! 2024-01-13T08:55:41.833 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-01-13T08:55:42.247 INFO:teuthology.orchestra.run.smithi141.stdout:Updating Subscription Management repositories. 2024-01-13T08:55:42.286 INFO:teuthology.orchestra.run.smithi077.stdout:Updating Subscription Management repositories. 2024-01-13T08:55:43.962 INFO:teuthology.orchestra.run.smithi141.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 34 kB/s | 4.1 kB 00:00 2024-01-13T08:55:44.040 INFO:teuthology.orchestra.run.smithi077.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 49 kB/s | 4.1 kB 00:00 2024-01-13T08:55:44.299 INFO:teuthology.orchestra.run.smithi141.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 56 kB/s | 4.5 kB 00:00 2024-01-13T08:55:44.377 INFO:teuthology.orchestra.run.smithi077.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 35 kB/s | 4.5 kB 00:00 2024-01-13T08:55:47.211 INFO:teuthology.orchestra.run.smithi141.stdout:Dependencies resolved. 2024-01-13T08:55:47.363 INFO:teuthology.orchestra.run.smithi141.stdout:========================================================================================================== 2024-01-13T08:55:47.364 INFO:teuthology.orchestra.run.smithi141.stdout: Package Arch Version Repository Size 2024-01-13T08:55:47.364 INFO:teuthology.orchestra.run.smithi141.stdout:========================================================================================================== 2024-01-13T08:55:47.364 INFO:teuthology.orchestra.run.smithi141.stdout:Installing group/module packages: 2024-01-13T08:55:47.364 INFO:teuthology.orchestra.run.smithi141.stdout: aardvark-dns x86_64 2:1.7.0-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 1.0 M 2024-01-13T08:55:47.364 INFO:teuthology.orchestra.run.smithi141.stdout: buildah x86_64 1:1.31.3-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 8.8 M 2024-01-13T08:55:47.364 INFO:teuthology.orchestra.run.smithi141.stdout: cockpit-podman noarch 75-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 739 k 2024-01-13T08:55:47.364 INFO:teuthology.orchestra.run.smithi141.stdout: crun x86_64 1.8.7-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 239 k 2024-01-13T08:55:47.364 INFO:teuthology.orchestra.run.smithi141.stdout: netavark x86_64 2:1.7.0-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-01-13T08:55:47.364 INFO:teuthology.orchestra.run.smithi141.stdout: python3-podman noarch 4.6.0-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 153 k 2024-01-13T08:55:47.364 INFO:teuthology.orchestra.run.smithi141.stdout: skopeo x86_64 2:1.13.3-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 8.1 M 2024-01-13T08:55:47.365 INFO:teuthology.orchestra.run.smithi141.stdout: toolbox x86_64 0.0.99.4-5.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 2.5 M 2024-01-13T08:55:47.365 INFO:teuthology.orchestra.run.smithi141.stdout: udica noarch 0.2.6-20.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 49 k 2024-01-13T08:55:47.365 INFO:teuthology.orchestra.run.smithi141.stdout:Installing dependencies: 2024-01-13T08:55:47.365 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2024-01-13T08:55:47.365 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2024-01-13T08:55:47.365 INFO:teuthology.orchestra.run.smithi141.stdout: yajl x86_64 2.1.0-12.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2024-01-13T08:55:47.365 INFO:teuthology.orchestra.run.smithi141.stdout:Installing weak dependencies: 2024-01-13T08:55:47.365 INFO:teuthology.orchestra.run.smithi141.stdout: criu-libs x86_64 3.18-4.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 39 k 2024-01-13T08:55:47.365 INFO:teuthology.orchestra.run.smithi141.stdout:Installing module profiles: 2024-01-13T08:55:47.365 INFO:teuthology.orchestra.run.smithi141.stdout: container-tools/common 2024-01-13T08:55:47.365 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-13T08:55:47.365 INFO:teuthology.orchestra.run.smithi141.stdout:Enabling module streams: 2024-01-13T08:55:47.366 INFO:teuthology.orchestra.run.smithi141.stdout: container-tools rhel8 2024-01-13T08:55:47.366 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-13T08:55:47.366 INFO:teuthology.orchestra.run.smithi141.stdout:Transaction Summary 2024-01-13T08:55:47.366 INFO:teuthology.orchestra.run.smithi141.stdout:========================================================================================================== 2024-01-13T08:55:47.366 INFO:teuthology.orchestra.run.smithi141.stdout:Install 13 Packages 2024-01-13T08:55:47.366 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-13T08:55:47.366 INFO:teuthology.orchestra.run.smithi141.stdout:Total download size: 26 M 2024-01-13T08:55:47.367 INFO:teuthology.orchestra.run.smithi141.stdout:Installed size: 87 M 2024-01-13T08:55:47.367 INFO:teuthology.orchestra.run.smithi141.stdout:Downloading Packages: 2024-01-13T08:55:47.563 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-01-13T08:55:47.592 INFO:teuthology.orchestra.run.smithi141.stdout:(1/13): python3-pytoml-0.1.14-5.git7dea353.el8. 110 kB/s | 25 kB 00:00 2024-01-13T08:55:47.618 INFO:teuthology.orchestra.run.smithi141.stdout:(2/13): python3-pyxdg-0.25-16.el8.noarch.rpm 374 kB/s | 94 kB 00:00 2024-01-13T08:55:47.660 INFO:teuthology.orchestra.run.smithi141.stdout:(3/13): yajl-2.1.0-12.el8.x86_64.rpm 140 kB/s | 41 kB 00:00 2024-01-13T08:55:47.716 INFO:teuthology.orchestra.run.smithi077.stdout:========================================================================================================== 2024-01-13T08:55:47.716 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2024-01-13T08:55:47.716 INFO:teuthology.orchestra.run.smithi077.stdout:========================================================================================================== 2024-01-13T08:55:47.716 INFO:teuthology.orchestra.run.smithi077.stdout:Installing group/module packages: 2024-01-13T08:55:47.717 INFO:teuthology.orchestra.run.smithi077.stdout: aardvark-dns x86_64 2:1.7.0-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 1.0 M 2024-01-13T08:55:47.717 INFO:teuthology.orchestra.run.smithi077.stdout: buildah x86_64 1:1.31.3-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 8.8 M 2024-01-13T08:55:47.717 INFO:teuthology.orchestra.run.smithi077.stdout: cockpit-podman noarch 75-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 739 k 2024-01-13T08:55:47.717 INFO:teuthology.orchestra.run.smithi077.stdout: crun x86_64 1.8.7-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 239 k 2024-01-13T08:55:47.717 INFO:teuthology.orchestra.run.smithi077.stdout: netavark x86_64 2:1.7.0-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-01-13T08:55:47.717 INFO:teuthology.orchestra.run.smithi077.stdout: python3-podman noarch 4.6.0-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 153 k 2024-01-13T08:55:47.717 INFO:teuthology.orchestra.run.smithi077.stdout: skopeo x86_64 2:1.13.3-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 8.1 M 2024-01-13T08:55:47.717 INFO:teuthology.orchestra.run.smithi077.stdout: toolbox x86_64 0.0.99.4-5.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 2.5 M 2024-01-13T08:55:47.717 INFO:teuthology.orchestra.run.smithi077.stdout: udica noarch 0.2.6-20.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 49 k 2024-01-13T08:55:47.717 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2024-01-13T08:55:47.717 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2024-01-13T08:55:47.717 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2024-01-13T08:55:47.718 INFO:teuthology.orchestra.run.smithi077.stdout: yajl x86_64 2.1.0-12.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2024-01-13T08:55:47.718 INFO:teuthology.orchestra.run.smithi077.stdout:Installing weak dependencies: 2024-01-13T08:55:47.718 INFO:teuthology.orchestra.run.smithi077.stdout: criu-libs x86_64 3.18-4.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 39 k 2024-01-13T08:55:47.718 INFO:teuthology.orchestra.run.smithi077.stdout:Installing module profiles: 2024-01-13T08:55:47.718 INFO:teuthology.orchestra.run.smithi077.stdout: container-tools/common 2024-01-13T08:55:47.718 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T08:55:47.718 INFO:teuthology.orchestra.run.smithi077.stdout:Enabling module streams: 2024-01-13T08:55:47.718 INFO:teuthology.orchestra.run.smithi077.stdout: container-tools rhel8 2024-01-13T08:55:47.718 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T08:55:47.718 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-01-13T08:55:47.718 INFO:teuthology.orchestra.run.smithi077.stdout:========================================================================================================== 2024-01-13T08:55:47.718 INFO:teuthology.orchestra.run.smithi077.stdout:Install 13 Packages 2024-01-13T08:55:47.718 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T08:55:47.719 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 26 M 2024-01-13T08:55:47.719 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 87 M 2024-01-13T08:55:47.719 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-01-13T08:55:47.823 INFO:teuthology.orchestra.run.smithi077.stdout:(1/13): yajl-2.1.0-12.el8.x86_64.rpm 393 kB/s | 41 kB 00:00 2024-01-13T08:55:47.849 INFO:teuthology.orchestra.run.smithi077.stdout:(2/13): python3-pyxdg-0.25-16.el8.noarch.rpm 725 kB/s | 94 kB 00:00 2024-01-13T08:55:47.860 INFO:teuthology.orchestra.run.smithi141.stdout:(4/13): aardvark-dns-1.7.0-1.module+el8.9.0+203 3.8 MB/s | 1.0 MB 00:00 2024-01-13T08:55:47.891 INFO:teuthology.orchestra.run.smithi077.stdout:(3/13): python3-pytoml-0.1.14-5.git7dea353.el8. 144 kB/s | 25 kB 00:00 2024-01-13T08:55:47.991 INFO:teuthology.orchestra.run.smithi077.stdout:(4/13): cockpit-podman-75-1.module+el8.9.0+2032 7.2 MB/s | 739 kB 00:00 2024-01-13T08:55:48.019 INFO:teuthology.orchestra.run.smithi141.stdout:(5/13): buildah-1.31.3-1.module+el8.9.0+20326+3 22 MB/s | 8.8 MB 00:00 2024-01-13T08:55:48.041 INFO:teuthology.orchestra.run.smithi077.stdout:(5/13): aardvark-dns-1.7.0-1.module+el8.9.0+203 4.6 MB/s | 1.0 MB 00:00 2024-01-13T08:55:48.044 INFO:teuthology.orchestra.run.smithi141.stdout:(6/13): criu-libs-3.18-4.module+el8.9.0+20326+3 214 kB/s | 39 kB 00:00 2024-01-13T08:55:48.078 INFO:teuthology.orchestra.run.smithi141.stdout:(7/13): cockpit-podman-75-1.module+el8.9.0+2032 1.7 MB/s | 739 kB 00:00 2024-01-13T08:55:48.208 INFO:teuthology.orchestra.run.smithi077.stdout:(6/13): buildah-1.31.3-1.module+el8.9.0+20326+3 25 MB/s | 8.8 MB 00:00 2024-01-13T08:55:48.233 INFO:teuthology.orchestra.run.smithi077.stdout:(7/13): criu-libs-3.18-4.module+el8.9.0+20326+3 162 kB/s | 39 kB 00:00 2024-01-13T08:55:48.236 INFO:teuthology.orchestra.run.smithi141.stdout:(8/13): crun-1.8.7-1.module+el8.9.0+20326+38708 1.1 MB/s | 239 kB 00:00 2024-01-13T08:55:48.259 INFO:teuthology.orchestra.run.smithi077.stdout:(8/13): crun-1.8.7-1.module+el8.9.0+20326+38708 1.1 MB/s | 239 kB 00:00 2024-01-13T08:55:48.303 INFO:teuthology.orchestra.run.smithi141.stdout:(9/13): netavark-1.7.0-1.module+el8.9.0+20326+3 14 MB/s | 3.7 MB 00:00 2024-01-13T08:55:48.412 INFO:teuthology.orchestra.run.smithi141.stdout:(10/13): python3-podman-4.6.0-1.module+el8.9.0+ 459 kB/s | 153 kB 00:00 2024-01-13T08:55:48.451 INFO:teuthology.orchestra.run.smithi077.stdout:(9/13): netavark-1.7.0-1.module+el8.9.0+20326+3 15 MB/s | 3.7 MB 00:00 2024-01-13T08:55:48.504 INFO:teuthology.orchestra.run.smithi141.stdout:(11/13): skopeo-1.13.3-1.module+el8.9.0+20326+3 30 MB/s | 8.1 MB 00:00 2024-01-13T08:55:48.621 INFO:teuthology.orchestra.run.smithi141.stdout:(12/13): toolbox-0.0.99.4-5.module+el8.9.0+2032 8.0 MB/s | 2.5 MB 00:00 2024-01-13T08:55:48.626 INFO:teuthology.orchestra.run.smithi077.stdout:(10/13): skopeo-1.13.3-1.module+el8.9.0+20326+3 22 MB/s | 8.1 MB 00:00 2024-01-13T08:55:48.651 INFO:teuthology.orchestra.run.smithi077.stdout:(11/13): python3-podman-4.6.0-1.module+el8.9.0+ 367 kB/s | 153 kB 00:00 2024-01-13T08:55:48.671 INFO:teuthology.orchestra.run.smithi141.stdout:(13/13): udica-0.2.6-20.module+el8.9.0+20326+38 191 kB/s | 49 kB 00:00 2024-01-13T08:55:48.671 INFO:teuthology.orchestra.run.smithi141.stdout:-------------------------------------------------------------------------------- 2024-01-13T08:55:48.671 INFO:teuthology.orchestra.run.smithi141.stdout:Total 20 MB/s | 26 MB 00:01 2024-01-13T08:55:48.710 INFO:teuthology.orchestra.run.smithi077.stdout:(12/13): toolbox-0.0.99.4-5.module+el8.9.0+2032 9.8 MB/s | 2.5 MB 00:00 2024-01-13T08:55:48.735 INFO:teuthology.orchestra.run.smithi077.stdout:(13/13): udica-0.2.6-20.module+el8.9.0+20326+38 456 kB/s | 49 kB 00:00 2024-01-13T08:55:48.735 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-01-13T08:55:48.735 INFO:teuthology.orchestra.run.smithi077.stdout:Total 25 MB/s | 26 MB 00:01 2024-01-13T08:55:48.949 INFO:teuthology.orchestra.run.smithi141.stdout:Running transaction check 2024-01-13T08:55:48.970 INFO:teuthology.orchestra.run.smithi141.stdout:Transaction check succeeded. 2024-01-13T08:55:48.971 INFO:teuthology.orchestra.run.smithi141.stdout:Running transaction test 2024-01-13T08:55:49.008 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-01-13T08:55:49.031 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-01-13T08:55:49.031 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-01-13T08:55:49.200 INFO:teuthology.orchestra.run.smithi141.stdout:Transaction test succeeded. 2024-01-13T08:55:49.209 INFO:teuthology.orchestra.run.smithi141.stdout:Running transaction 2024-01-13T08:55:49.257 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-01-13T08:55:49.266 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-01-13T08:55:49.797 INFO:teuthology.orchestra.run.smithi141.stdout: Preparing : 1/1 2024-01-13T08:55:49.867 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-01-13T08:55:50.043 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 1/13 2024-01-13T08:55:50.147 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 1/13 2024-01-13T08:55:50.569 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : aardvark-dns-2:1.7.0-1.module+el8.9.0+20326+387084 2/13 2024-01-13T08:55:50.660 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : aardvark-dns-2:1.7.0-1.module+el8.9.0+20326+387084 2/13 2024-01-13T08:55:50.779 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : netavark-2:1.7.0-1.module+el8.9.0+20326+387084d0.x 3/13 2024-01-13T08:55:50.893 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : netavark-2:1.7.0-1.module+el8.9.0+20326+387084d0.x 3/13 2024-01-13T08:55:50.978 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : yajl-2.1.0-12.el8.x86_64 4/13 2024-01-13T08:55:51.115 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : yajl-2.1.0-12.el8.x86_64 4/13 2024-01-13T08:55:51.157 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 5/13 2024-01-13T08:55:51.327 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 5/13 2024-01-13T08:55:51.435 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 6/13 2024-01-13T08:55:51.588 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 6/13 2024-01-13T08:55:51.659 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : python3-podman-4.6.0-1.module+el8.9.0+20326+387084 7/13 2024-01-13T08:55:51.822 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-podman-4.6.0-1.module+el8.9.0+20326+387084 7/13 2024-01-13T08:55:52.557 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : crun-1.8.7-1.module+el8.9.0+20326+387084d0.x86_64 8/13 2024-01-13T08:55:52.762 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : crun-1.8.7-1.module+el8.9.0+20326+387084d0.x86_64 8/13 2024-01-13T08:55:52.767 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : buildah-1:1.31.3-1.module+el8.9.0+20326+387084d0.x 9/13 2024-01-13T08:55:53.049 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : buildah-1:1.31.3-1.module+el8.9.0+20326+387084d0.x 9/13 2024-01-13T08:55:53.185 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : udica-0.2.6-20.module+el8.9.0+20326+387084d0.noarc 10/13 2024-01-13T08:55:53.441 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : udica-0.2.6-20.module+el8.9.0+20326+387084d0.noarc 10/13 2024-01-13T08:55:54.090 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : toolbox-0.0.99.4-5.module+el8.9.0+20326+387084d0.x 11/13 2024-01-13T08:55:54.336 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : toolbox-0.0.99.4-5.module+el8.9.0+20326+387084d0.x 11/13 2024-01-13T08:55:54.337 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : skopeo-2:1.13.3-1.module+el8.9.0+20326+387084d0.x8 12/13 2024-01-13T08:55:54.588 INFO:teuthology.orchestra.run.smithi141.stdout: Installing : cockpit-podman-75-1.module+el8.9.0+20326+387084d0. 13/13 2024-01-13T08:55:54.609 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : skopeo-2:1.13.3-1.module+el8.9.0+20326+387084d0.x8 12/13 2024-01-13T08:55:54.800 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cockpit-podman-75-1.module+el8.9.0+20326+387084d0. 13/13 2024-01-13T08:55:54.998 INFO:teuthology.orchestra.run.smithi141.stdout: Running scriptlet: cockpit-podman-75-1.module+el8.9.0+20326+387084d0. 13/13 2024-01-13T08:55:54.998 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/13 2024-01-13T08:55:54.999 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-01-13T08:55:54.999 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 3/13 2024-01-13T08:55:54.999 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : aardvark-dns-2:1.7.0-1.module+el8.9.0+20326+387084 4/13 2024-01-13T08:55:54.999 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : buildah-1:1.31.3-1.module+el8.9.0+20326+387084d0.x 5/13 2024-01-13T08:55:54.999 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : cockpit-podman-75-1.module+el8.9.0+20326+387084d0. 6/13 2024-01-13T08:55:54.999 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 7/13 2024-01-13T08:55:54.999 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : crun-1.8.7-1.module+el8.9.0+20326+387084d0.x86_64 8/13 2024-01-13T08:55:54.999 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : netavark-2:1.7.0-1.module+el8.9.0+20326+387084d0.x 9/13 2024-01-13T08:55:54.999 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : python3-podman-4.6.0-1.module+el8.9.0+20326+387084 10/13 2024-01-13T08:55:54.999 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : skopeo-2:1.13.3-1.module+el8.9.0+20326+387084d0.x8 11/13 2024-01-13T08:55:54.999 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : toolbox-0.0.99.4-5.module+el8.9.0+20326+387084d0.x 12/13 2024-01-13T08:55:55.252 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: cockpit-podman-75-1.module+el8.9.0+20326+387084d0. 13/13 2024-01-13T08:55:55.252 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/13 2024-01-13T08:55:55.252 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-01-13T08:55:55.252 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 3/13 2024-01-13T08:55:55.252 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : aardvark-dns-2:1.7.0-1.module+el8.9.0+20326+387084 4/13 2024-01-13T08:55:55.252 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : buildah-1:1.31.3-1.module+el8.9.0+20326+387084d0.x 5/13 2024-01-13T08:55:55.252 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cockpit-podman-75-1.module+el8.9.0+20326+387084d0. 6/13 2024-01-13T08:55:55.252 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 7/13 2024-01-13T08:55:55.252 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : crun-1.8.7-1.module+el8.9.0+20326+387084d0.x86_64 8/13 2024-01-13T08:55:55.252 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : netavark-2:1.7.0-1.module+el8.9.0+20326+387084d0.x 9/13 2024-01-13T08:55:55.252 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-podman-4.6.0-1.module+el8.9.0+20326+387084 10/13 2024-01-13T08:55:55.252 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : skopeo-2:1.13.3-1.module+el8.9.0+20326+387084d0.x8 11/13 2024-01-13T08:55:55.253 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : toolbox-0.0.99.4-5.module+el8.9.0+20326+387084d0.x 12/13 2024-01-13T08:56:06.750 INFO:teuthology.orchestra.run.smithi141.stdout: Verifying : udica-0.2.6-20.module+el8.9.0+20326+387084d0.noarc 13/13 2024-01-13T08:56:06.751 INFO:teuthology.orchestra.run.smithi141.stdout:Installed products updated. 2024-01-13T08:56:07.277 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-13T08:56:07.277 INFO:teuthology.orchestra.run.smithi141.stdout:Installed: 2024-01-13T08:56:07.277 INFO:teuthology.orchestra.run.smithi141.stdout: aardvark-dns-2:1.7.0-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-13T08:56:07.277 INFO:teuthology.orchestra.run.smithi141.stdout: buildah-1:1.31.3-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-13T08:56:07.277 INFO:teuthology.orchestra.run.smithi141.stdout: cockpit-podman-75-1.module+el8.9.0+20326+387084d0.noarch 2024-01-13T08:56:07.278 INFO:teuthology.orchestra.run.smithi141.stdout: criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86_64 2024-01-13T08:56:07.278 INFO:teuthology.orchestra.run.smithi141.stdout: crun-1.8.7-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-13T08:56:07.278 INFO:teuthology.orchestra.run.smithi141.stdout: netavark-2:1.7.0-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-13T08:56:07.278 INFO:teuthology.orchestra.run.smithi141.stdout: python3-podman-4.6.0-1.module+el8.9.0+20326+387084d0.noarch 2024-01-13T08:56:07.278 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-01-13T08:56:07.278 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-01-13T08:56:07.278 INFO:teuthology.orchestra.run.smithi141.stdout: skopeo-2:1.13.3-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-13T08:56:07.278 INFO:teuthology.orchestra.run.smithi141.stdout: toolbox-0.0.99.4-5.module+el8.9.0+20326+387084d0.x86_64 2024-01-13T08:56:07.278 INFO:teuthology.orchestra.run.smithi141.stdout: udica-0.2.6-20.module+el8.9.0+20326+387084d0.noarch 2024-01-13T08:56:07.278 INFO:teuthology.orchestra.run.smithi141.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-13T08:56:07.278 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-13T08:56:07.278 INFO:teuthology.orchestra.run.smithi141.stdout:Complete! 2024-01-13T08:56:07.455 DEBUG:teuthology.parallel:result is None 2024-01-13T08:56:08.076 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : udica-0.2.6-20.module+el8.9.0+20326+387084d0.noarc 13/13 2024-01-13T08:56:08.076 INFO:teuthology.orchestra.run.smithi077.stdout:Installed products updated. 2024-01-13T08:56:08.630 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T08:56:08.630 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-01-13T08:56:08.630 INFO:teuthology.orchestra.run.smithi077.stdout: aardvark-dns-2:1.7.0-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-13T08:56:08.630 INFO:teuthology.orchestra.run.smithi077.stdout: buildah-1:1.31.3-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-13T08:56:08.630 INFO:teuthology.orchestra.run.smithi077.stdout: cockpit-podman-75-1.module+el8.9.0+20326+387084d0.noarch 2024-01-13T08:56:08.630 INFO:teuthology.orchestra.run.smithi077.stdout: criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86_64 2024-01-13T08:56:08.630 INFO:teuthology.orchestra.run.smithi077.stdout: crun-1.8.7-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-13T08:56:08.630 INFO:teuthology.orchestra.run.smithi077.stdout: netavark-2:1.7.0-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-13T08:56:08.630 INFO:teuthology.orchestra.run.smithi077.stdout: python3-podman-4.6.0-1.module+el8.9.0+20326+387084d0.noarch 2024-01-13T08:56:08.630 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-01-13T08:56:08.631 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-01-13T08:56:08.631 INFO:teuthology.orchestra.run.smithi077.stdout: skopeo-2:1.13.3-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-13T08:56:08.631 INFO:teuthology.orchestra.run.smithi077.stdout: toolbox-0.0.99.4-5.module+el8.9.0+20326+387084d0.x86_64 2024-01-13T08:56:08.631 INFO:teuthology.orchestra.run.smithi077.stdout: udica-0.2.6-20.module+el8.9.0+20326+387084d0.noarch 2024-01-13T08:56:08.631 INFO:teuthology.orchestra.run.smithi077.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-13T08:56:08.631 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T08:56:08.631 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-01-13T08:56:08.799 DEBUG:teuthology.parallel:result is None 2024-01-13T08:56:08.799 INFO:teuthology.run_tasks:Running task nvme_loop... 2024-01-13T08:56:08.811 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2024-01-13T08:56:08.811 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-13T08:56:08.812 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/scratch_devs of=/dev/stdout 2024-01-13T08:56:08.830 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-13T08:56:08.830 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_1 2024-01-13T08:56:08.887 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-13T08:56:08.887 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-13T08:56:08.888 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 6h/6d Inode: 169327 Links: 1 2024-01-13T08:56:08.888 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-13T08:56:08.888 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-01-13T08:56:08.888 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-01-13 08:55:54.991280041 +0000 2024-01-13T08:56:08.888 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-01-13 08:54:46.207847217 +0000 2024-01-13T08:56:08.888 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-01-13 08:54:46.207847217 +0000 2024-01-13T08:56:08.888 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2024-01-13T08:56:08.888 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-13T08:56:08.953 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-01-13T08:56:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-01-13T08:56:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000273199 s, 1.9 MB/s 2024-01-13T08:56:08.955 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-13T08:56:09.012 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_2 2024-01-13T08:56:09.070 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-13T08:56:09.070 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-13T08:56:09.071 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 6h/6d Inode: 168586 Links: 1 2024-01-13T08:56:09.071 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-13T08:56:09.071 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-01-13T08:56:09.071 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-01-13 08:55:54.991280041 +0000 2024-01-13T08:56:09.071 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-01-13 08:54:46.576838994 +0000 2024-01-13T08:56:09.071 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-01-13 08:54:46.576838994 +0000 2024-01-13T08:56:09.071 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2024-01-13T08:56:09.071 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-13T08:56:09.136 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-01-13T08:56:09.136 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-01-13T08:56:09.136 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000263462 s, 1.9 MB/s 2024-01-13T08:56:09.138 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-13T08:56:09.196 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_3 2024-01-13T08:56:09.253 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-13T08:56:09.254 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-13T08:56:09.254 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 6h/6d Inode: 165758 Links: 1 2024-01-13T08:56:09.254 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-13T08:56:09.254 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-01-13T08:56:09.254 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-01-13 08:55:54.991280041 +0000 2024-01-13T08:56:09.254 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-01-13 08:54:46.935830993 +0000 2024-01-13T08:56:09.254 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-01-13 08:54:46.935830993 +0000 2024-01-13T08:56:09.254 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2024-01-13T08:56:09.254 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-13T08:56:09.318 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-01-13T08:56:09.318 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-01-13T08:56:09.318 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000192194 s, 2.7 MB/s 2024-01-13T08:56:09.319 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-13T08:56:09.378 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_4 2024-01-13T08:56:09.435 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-13T08:56:09.436 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-13T08:56:09.436 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 6h/6d Inode: 169452 Links: 1 2024-01-13T08:56:09.436 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-13T08:56:09.436 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-01-13T08:56:09.436 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-01-13 08:55:54.991280041 +0000 2024-01-13T08:56:09.436 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-01-13 08:54:47.261823727 +0000 2024-01-13T08:56:09.436 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-01-13 08:54:47.261823727 +0000 2024-01-13T08:56:09.436 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2024-01-13T08:56:09.436 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-13T08:56:09.500 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-01-13T08:56:09.500 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-01-13T08:56:09.501 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000236619 s, 2.2 MB/s 2024-01-13T08:56:09.501 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-13T08:56:09.559 DEBUG:teuthology.orchestra.run.smithi077:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-01-13T08:56:09.688 INFO:teuthology.orchestra.run.smithi077.stdout:loop 2024-01-13T08:56:09.689 INFO:tasks.nvme_loop:Connecting nvme_loop smithi077:/dev/vg_nvme/lv_1... 2024-01-13T08:56:09.689 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-01-13T08:56:09.725 INFO:teuthology.orchestra.run.smithi077.stdout:1 2024-01-13T08:56:09.761 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/vg_nvme/lv_11 2024-01-13T08:56:09.802 INFO:tasks.nvme_loop:Connecting nvme_loop smithi077:/dev/vg_nvme/lv_2... 2024-01-13T08:56:09.802 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-01-13T08:56:09.879 INFO:teuthology.orchestra.run.smithi077.stdout:1 2024-01-13T08:56:09.915 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/vg_nvme/lv_21 2024-01-13T08:56:09.953 INFO:tasks.nvme_loop:Connecting nvme_loop smithi077:/dev/vg_nvme/lv_3... 2024-01-13T08:56:09.954 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-01-13T08:56:10.029 INFO:teuthology.orchestra.run.smithi077.stdout:1 2024-01-13T08:56:10.065 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/vg_nvme/lv_31 2024-01-13T08:56:10.104 INFO:tasks.nvme_loop:Connecting nvme_loop smithi077:/dev/vg_nvme/lv_4... 2024-01-13T08:56:10.104 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-01-13T08:56:10.184 INFO:teuthology.orchestra.run.smithi077.stdout:1 2024-01-13T08:56:10.220 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/vg_nvme/lv_41 2024-01-13T08:56:10.259 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-13T08:56:10.259 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/scratch_devs of=/dev/stdout 2024-01-13T08:56:10.316 DEBUG:teuthology.orchestra.run.smithi077:> sudo nvme list 2024-01-13T08:56:10.382 INFO:teuthology.orchestra.run.smithi077.stdout:Node SN Model Namespace Usage Format FW Rev 2024-01-13T08:56:10.382 INFO:teuthology.orchestra.run.smithi077.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-01-13T08:56:10.382 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/nvme0n1 PHFT620400ZJ400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-01-13T08:56:10.382 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/nvme1n1 9fccf3a6479b395c383a Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2024-01-13T08:56:10.382 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/nvme2n1 b6187ff4dc3646014814 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2024-01-13T08:56:10.382 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/nvme3n1 e9aee6e36ad978fcb3ca Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2024-01-13T08:56:10.382 INFO:teuthology.orchestra.run.smithi077.stdout:/dev/nvme4n1 25898a8d04bbd2b3fde9 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2024-01-13T08:56:10.383 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-01-13T08:56:10.383 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-13T08:56:10.383 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/scratch_devs 2024-01-13T08:56:10.450 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-13T08:56:10.450 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/scratch_devs of=/dev/stdout 2024-01-13T08:56:10.472 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-13T08:56:10.473 DEBUG:teuthology.orchestra.run.smithi141:> stat /dev/vg_nvme/lv_1 2024-01-13T08:56:10.532 INFO:teuthology.orchestra.run.smithi141.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-13T08:56:10.532 INFO:teuthology.orchestra.run.smithi141.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-13T08:56:10.533 INFO:teuthology.orchestra.run.smithi141.stdout:Device: 6h/6d Inode: 170516 Links: 1 2024-01-13T08:56:10.533 INFO:teuthology.orchestra.run.smithi141.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-13T08:56:10.533 INFO:teuthology.orchestra.run.smithi141.stdout:Context: system_u:object_r:device_t:s0 2024-01-13T08:56:10.533 INFO:teuthology.orchestra.run.smithi141.stdout:Access: 2024-01-13 08:55:54.738418393 +0000 2024-01-13T08:56:10.533 INFO:teuthology.orchestra.run.smithi141.stdout:Modify: 2024-01-13 08:54:51.355996573 +0000 2024-01-13T08:56:10.533 INFO:teuthology.orchestra.run.smithi141.stdout:Change: 2024-01-13 08:54:51.355996573 +0000 2024-01-13T08:56:10.533 INFO:teuthology.orchestra.run.smithi141.stdout: Birth: - 2024-01-13T08:56:10.533 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-13T08:56:10.602 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records in 2024-01-13T08:56:10.603 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records out 2024-01-13T08:56:10.603 INFO:teuthology.orchestra.run.smithi141.stderr:512 bytes copied, 0.000280187 s, 1.8 MB/s 2024-01-13T08:56:10.604 DEBUG:teuthology.orchestra.run.smithi141:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-13T08:56:10.664 DEBUG:teuthology.orchestra.run.smithi141:> stat /dev/vg_nvme/lv_2 2024-01-13T08:56:10.723 INFO:teuthology.orchestra.run.smithi141.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-13T08:56:10.724 INFO:teuthology.orchestra.run.smithi141.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-13T08:56:10.724 INFO:teuthology.orchestra.run.smithi141.stdout:Device: 6h/6d Inode: 170560 Links: 1 2024-01-13T08:56:10.724 INFO:teuthology.orchestra.run.smithi141.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-13T08:56:10.724 INFO:teuthology.orchestra.run.smithi141.stdout:Context: system_u:object_r:device_t:s0 2024-01-13T08:56:10.724 INFO:teuthology.orchestra.run.smithi141.stdout:Access: 2024-01-13 08:55:54.738418393 +0000 2024-01-13T08:56:10.724 INFO:teuthology.orchestra.run.smithi141.stdout:Modify: 2024-01-13 08:54:51.704989224 +0000 2024-01-13T08:56:10.724 INFO:teuthology.orchestra.run.smithi141.stdout:Change: 2024-01-13 08:54:51.704989224 +0000 2024-01-13T08:56:10.724 INFO:teuthology.orchestra.run.smithi141.stdout: Birth: - 2024-01-13T08:56:10.724 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-13T08:56:10.795 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records in 2024-01-13T08:56:10.795 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records out 2024-01-13T08:56:10.795 INFO:teuthology.orchestra.run.smithi141.stderr:512 bytes copied, 0.000236167 s, 2.2 MB/s 2024-01-13T08:56:10.796 DEBUG:teuthology.orchestra.run.smithi141:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-13T08:56:10.856 DEBUG:teuthology.orchestra.run.smithi141:> stat /dev/vg_nvme/lv_3 2024-01-13T08:56:10.914 INFO:teuthology.orchestra.run.smithi141.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-13T08:56:10.914 INFO:teuthology.orchestra.run.smithi141.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-13T08:56:10.914 INFO:teuthology.orchestra.run.smithi141.stdout:Device: 6h/6d Inode: 168507 Links: 1 2024-01-13T08:56:10.914 INFO:teuthology.orchestra.run.smithi141.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-13T08:56:10.914 INFO:teuthology.orchestra.run.smithi141.stdout:Context: system_u:object_r:device_t:s0 2024-01-13T08:56:10.915 INFO:teuthology.orchestra.run.smithi141.stdout:Access: 2024-01-13 08:55:54.738418393 +0000 2024-01-13T08:56:10.915 INFO:teuthology.orchestra.run.smithi141.stdout:Modify: 2024-01-13 08:54:52.056981812 +0000 2024-01-13T08:56:10.915 INFO:teuthology.orchestra.run.smithi141.stdout:Change: 2024-01-13 08:54:52.056981812 +0000 2024-01-13T08:56:10.915 INFO:teuthology.orchestra.run.smithi141.stdout: Birth: - 2024-01-13T08:56:10.915 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-13T08:56:10.982 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records in 2024-01-13T08:56:10.982 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records out 2024-01-13T08:56:10.982 INFO:teuthology.orchestra.run.smithi141.stderr:512 bytes copied, 0.000209331 s, 2.4 MB/s 2024-01-13T08:56:10.983 DEBUG:teuthology.orchestra.run.smithi141:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-13T08:56:11.043 DEBUG:teuthology.orchestra.run.smithi141:> stat /dev/vg_nvme/lv_4 2024-01-13T08:56:11.101 INFO:teuthology.orchestra.run.smithi141.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-13T08:56:11.101 INFO:teuthology.orchestra.run.smithi141.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-13T08:56:11.101 INFO:teuthology.orchestra.run.smithi141.stdout:Device: 6h/6d Inode: 170626 Links: 1 2024-01-13T08:56:11.101 INFO:teuthology.orchestra.run.smithi141.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-13T08:56:11.101 INFO:teuthology.orchestra.run.smithi141.stdout:Context: system_u:object_r:device_t:s0 2024-01-13T08:56:11.101 INFO:teuthology.orchestra.run.smithi141.stdout:Access: 2024-01-13 08:55:54.738418393 +0000 2024-01-13T08:56:11.101 INFO:teuthology.orchestra.run.smithi141.stdout:Modify: 2024-01-13 08:54:52.400974569 +0000 2024-01-13T08:56:11.101 INFO:teuthology.orchestra.run.smithi141.stdout:Change: 2024-01-13 08:54:52.400974569 +0000 2024-01-13T08:56:11.102 INFO:teuthology.orchestra.run.smithi141.stdout: Birth: - 2024-01-13T08:56:11.102 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-13T08:56:11.169 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records in 2024-01-13T08:56:11.169 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records out 2024-01-13T08:56:11.169 INFO:teuthology.orchestra.run.smithi141.stderr:512 bytes copied, 0.000249532 s, 2.1 MB/s 2024-01-13T08:56:11.170 DEBUG:teuthology.orchestra.run.smithi141:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-13T08:56:11.229 DEBUG:teuthology.orchestra.run.smithi141:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-01-13T08:56:11.374 INFO:teuthology.orchestra.run.smithi141.stdout:loop 2024-01-13T08:56:11.379 INFO:tasks.nvme_loop:Connecting nvme_loop smithi141:/dev/vg_nvme/lv_1... 2024-01-13T08:56:11.380 DEBUG:teuthology.orchestra.run.smithi141:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-01-13T08:56:11.415 INFO:teuthology.orchestra.run.smithi141.stdout:1 2024-01-13T08:56:11.450 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/vg_nvme/lv_11 2024-01-13T08:56:11.491 INFO:tasks.nvme_loop:Connecting nvme_loop smithi141:/dev/vg_nvme/lv_2... 2024-01-13T08:56:11.491 DEBUG:teuthology.orchestra.run.smithi141:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-01-13T08:56:11.569 INFO:teuthology.orchestra.run.smithi141.stdout:1 2024-01-13T08:56:11.604 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/vg_nvme/lv_21 2024-01-13T08:56:11.643 INFO:tasks.nvme_loop:Connecting nvme_loop smithi141:/dev/vg_nvme/lv_3... 2024-01-13T08:56:11.643 DEBUG:teuthology.orchestra.run.smithi141:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-01-13T08:56:11.717 INFO:teuthology.orchestra.run.smithi141.stdout:1 2024-01-13T08:56:11.753 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/vg_nvme/lv_31 2024-01-13T08:56:11.792 INFO:tasks.nvme_loop:Connecting nvme_loop smithi141:/dev/vg_nvme/lv_4... 2024-01-13T08:56:11.792 DEBUG:teuthology.orchestra.run.smithi141:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-01-13T08:56:11.868 INFO:teuthology.orchestra.run.smithi141.stdout:1 2024-01-13T08:56:11.904 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/vg_nvme/lv_41 2024-01-13T08:56:11.943 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-13T08:56:11.943 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/scratch_devs of=/dev/stdout 2024-01-13T08:56:12.001 DEBUG:teuthology.orchestra.run.smithi141:> sudo nvme list 2024-01-13T08:56:12.067 INFO:teuthology.orchestra.run.smithi141.stdout:Node SN Model Namespace Usage Format FW Rev 2024-01-13T08:56:12.067 INFO:teuthology.orchestra.run.smithi141.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-01-13T08:56:12.067 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/nvme0n1 CVFT6242001S400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-01-13T08:56:12.067 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/nvme1n1 b854a7493e634c407411 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2024-01-13T08:56:12.067 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/nvme2n1 ab85b30ec1af74a021f7 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2024-01-13T08:56:12.067 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/nvme3n1 4333fa6d09aafbe00564 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2024-01-13T08:56:12.067 INFO:teuthology.orchestra.run.smithi141.stdout:/dev/nvme4n1 ee29c92adaab92595985 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2024-01-13T08:56:12.068 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-01-13T08:56:12.068 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-13T08:56:12.069 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd of=/scratch_devs 2024-01-13T08:56:12.135 INFO:teuthology.run_tasks:Running task cephadm... 2024-01-13T08:56:12.215 INFO:tasks.cephadm:Config: {'roleless': True, 'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '445562ab4bc3ddfb386936119050695810860bcb'} 2024-01-13T08:56:12.215 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb 2024-01-13T08:56:12.216 INFO:tasks.cephadm:Cluster fsid is 99e41a38-b1f1-11ee-95ac-87774f69a715 2024-01-13T08:56:12.216 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-01-13T08:56:12.216 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-01-13T08:56:12.216 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi077': '172.21.15.77', 'mon.smithi141': '172.21.15.141'} 2024-01-13T08:56:12.216 INFO:tasks.cephadm:Normalizing hostnames... 2024-01-13T08:56:12.216 DEBUG:teuthology.orchestra.run.smithi077:> sudo hostname $(hostname -s) 2024-01-13T08:56:12.250 DEBUG:teuthology.orchestra.run.smithi141:> sudo hostname $(hostname -s) 2024-01-13T08:56:12.277 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph-ci.git ref 445562ab4bc3ddfb386936119050695810860bcb)... 2024-01-13T08:56:12.278 DEBUG:teuthology.orchestra.run.smithi077:> curl --silent https://raw.githubusercontent.com/ceph/ceph-ci/445562ab4bc3ddfb386936119050695810860bcb/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-13T08:56:12.639 INFO:teuthology.orchestra.run.smithi077.stdout:-rw-rw-r--. 1 ubuntu ubuntu 355729 Jan 13 08:56 /home/ubuntu/cephtest/cephadm 2024-01-13T08:56:12.639 DEBUG:teuthology.orchestra.run.smithi141:> curl --silent https://raw.githubusercontent.com/ceph/ceph-ci/445562ab4bc3ddfb386936119050695810860bcb/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-13T08:56:12.906 INFO:teuthology.orchestra.run.smithi141.stdout:-rw-rw-r--. 1 ubuntu ubuntu 355729 Jan 13 08:56 /home/ubuntu/cephtest/cephadm 2024-01-13T08:56:12.906 DEBUG:teuthology.orchestra.run.smithi077:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-13T08:56:12.927 DEBUG:teuthology.orchestra.run.smithi141:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-13T08:56:12.954 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /etc/ceph 2024-01-13T08:56:13.001 DEBUG:teuthology.orchestra.run.smithi141:> sudo mkdir -p /etc/ceph 2024-01-13T08:56:13.029 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 777 /etc/ceph 2024-01-13T08:56:13.072 DEBUG:teuthology.orchestra.run.smithi141:> sudo chmod 777 /etc/ceph 2024-01-13T08:56:13.098 INFO:tasks.cephadm:Writing seed config... 2024-01-13T08:56:13.099 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-01-13T08:56:13.100 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-01-13T08:56:13.100 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-01-13T08:56:13.100 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-01-13T08:56:13.100 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-01-13T08:56:13.100 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-01-13T08:56:13.100 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-01-13T08:56:13.100 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-01-13T08:56:13.100 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-13T08:56:13.101 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-01-13T08:56:13.131 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 99e41a38-b1f1-11ee-95ac-87774f69a715 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-01-13T08:56:13.131 DEBUG:teuthology.orchestra.run.smithi077:mon.smithi077> sudo journalctl -f -n 0 -u ceph-99e41a38-b1f1-11ee-95ac-87774f69a715@mon.smithi077.service 2024-01-13T08:56:13.173 INFO:tasks.cephadm:Bootstrapping... 2024-01-13T08:56:13.174 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb -v bootstrap --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.77 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-01-13T08:56:13.202 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:-- Logs begin at Sat 2024-01-13 08:46:22 UTC. -- 2024-01-13T08:56:13.458 INFO:teuthology.orchestra.run.smithi077.stderr:-------------------------------------------------------------------------------- 2024-01-13T08:56:13.458 INFO:teuthology.orchestra.run.smithi077.stderr:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb', '-v', 'bootstrap', '--fsid', '99e41a38-b1f1-11ee-95ac-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.77', '--skip-admin-label'] 2024-01-13T08:56:15.153 INFO:teuthology.orchestra.run.smithi077.stderr:/bin/podman: 4.6.1 2024-01-13T08:56:15.165 INFO:teuthology.orchestra.run.smithi077.stderr:Verifying podman|docker is present... 2024-01-13T08:56:15.188 INFO:teuthology.orchestra.run.smithi077.stderr:/bin/podman: 4.6.1 2024-01-13T08:56:15.197 INFO:teuthology.orchestra.run.smithi077.stderr:Verifying lvm2 is present... 2024-01-13T08:56:15.197 INFO:teuthology.orchestra.run.smithi077.stderr:Verifying time synchronization is in place... 2024-01-13T08:56:15.204 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-01-13T08:56:15.210 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: inactive 2024-01-13T08:56:15.217 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: enabled 2024-01-13T08:56:15.223 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: active 2024-01-13T08:56:15.224 INFO:teuthology.orchestra.run.smithi077.stderr:Unit chronyd.service is enabled and running 2024-01-13T08:56:15.224 INFO:teuthology.orchestra.run.smithi077.stderr:Repeating the final host check... 2024-01-13T08:56:15.245 INFO:teuthology.orchestra.run.smithi077.stderr:/bin/podman: 4.6.1 2024-01-13T08:56:15.257 INFO:teuthology.orchestra.run.smithi077.stderr:podman (/bin/podman) version 4.6.1 is present 2024-01-13T08:56:15.257 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl is present 2024-01-13T08:56:15.258 INFO:teuthology.orchestra.run.smithi077.stderr:lvcreate is present 2024-01-13T08:56:15.263 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-01-13T08:56:15.271 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: inactive 2024-01-13T08:56:15.275 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: enabled 2024-01-13T08:56:15.281 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: active 2024-01-13T08:56:15.282 INFO:teuthology.orchestra.run.smithi077.stderr:Unit chronyd.service is enabled and running 2024-01-13T08:56:15.282 INFO:teuthology.orchestra.run.smithi077.stderr:Host looks OK 2024-01-13T08:56:15.282 INFO:teuthology.orchestra.run.smithi077.stderr:Cluster fsid: 99e41a38-b1f1-11ee-95ac-87774f69a715 2024-01-13T08:56:15.282 INFO:teuthology.orchestra.run.smithi077.stderr:Acquiring lock 139695430332544 on /run/cephadm/99e41a38-b1f1-11ee-95ac-87774f69a715.lock 2024-01-13T08:56:15.283 INFO:teuthology.orchestra.run.smithi077.stderr:Lock 139695430332544 acquired on /run/cephadm/99e41a38-b1f1-11ee-95ac-87774f69a715.lock 2024-01-13T08:56:15.283 INFO:teuthology.orchestra.run.smithi077.stderr:Verifying IP 172.21.15.77 port 3300 ... 2024-01-13T08:56:15.283 INFO:teuthology.orchestra.run.smithi077.stderr:Verifying IP 172.21.15.77 port 6789 ... 2024-01-13T08:56:15.283 INFO:teuthology.orchestra.run.smithi077.stderr:Base mon IP(s) is [172.21.15.77:3300, 172.21.15.77:6789], mon addrv is [v2:172.21.15.77:3300,v1:172.21.15.77:6789] 2024-01-13T08:56:15.286 INFO:teuthology.orchestra.run.smithi077.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp metric 100 2024-01-13T08:56:15.286 INFO:teuthology.orchestra.run.smithi077.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.77 metric 100 2024-01-13T08:56:15.289 INFO:teuthology.orchestra.run.smithi077.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-01-13T08:56:15.289 INFO:teuthology.orchestra.run.smithi077.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-01-13T08:56:15.289 INFO:teuthology.orchestra.run.smithi077.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-01-13T08:56:15.292 INFO:teuthology.orchestra.run.smithi077.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-01-13T08:56:15.292 INFO:teuthology.orchestra.run.smithi077.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-01-13T08:56:15.292 INFO:teuthology.orchestra.run.smithi077.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-01-13T08:56:15.292 INFO:teuthology.orchestra.run.smithi077.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-01-13T08:56:15.292 INFO:teuthology.orchestra.run.smithi077.stderr:/sbin/ip: inet6 fe80::5dc9:5dea:297f:8709/64 scope link noprefixroute 2024-01-13T08:56:15.292 INFO:teuthology.orchestra.run.smithi077.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-01-13T08:56:15.293 INFO:teuthology.orchestra.run.smithi077.stderr:Mon IP `172.21.15.77` is in CIDR network `172.21.0.0/20` 2024-01-13T08:56:15.293 INFO:teuthology.orchestra.run.smithi077.stderr:Mon IP `172.21.15.77` is in CIDR network `172.21.0.0/20` 2024-01-13T08:56:15.293 INFO:teuthology.orchestra.run.smithi077.stderr:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-01-13T08:56:15.293 INFO:teuthology.orchestra.run.smithi077.stderr:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-01-13T08:56:15.294 INFO:teuthology.orchestra.run.smithi077.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb... 2024-01-13T08:56:15.411 INFO:teuthology.orchestra.run.smithi077.stderr:/bin/podman: Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb... 2024-01-13T08:56:15.489 INFO:teuthology.orchestra.run.smithi077.stderr:/bin/podman: Getting image source signatures 2024-01-13T08:56:15.490 INFO:teuthology.orchestra.run.smithi077.stderr:/bin/podman: Copying blob sha256:9bc82cfe809169b39ceaa8362ba30ca8525e0ac28e35f12009a7a395fd81b5b0 2024-01-13T08:56:15.490 INFO:teuthology.orchestra.run.smithi077.stderr:/bin/podman: Copying blob sha256:6c507150f7e9f028ef978d633e702dab944bbda82bfbc46f6307495cf6dd9fc7 2024-01-13T08:56:37.720 INFO:teuthology.orchestra.run.smithi077.stderr:/bin/podman: Copying config sha256:163fd1944c31b631006e5d238574cf3311ed176b42b55b0b64e2387958260fc0 2024-01-13T08:56:37.811 INFO:teuthology.orchestra.run.smithi077.stderr:/bin/podman: Writing manifest to image destination 2024-01-13T08:56:38.098 INFO:teuthology.orchestra.run.smithi077.stderr:/bin/podman: 163fd1944c31b631006e5d238574cf3311ed176b42b55b0b64e2387958260fc0 2024-01-13T08:56:38.637 INFO:teuthology.orchestra.run.smithi077.stderr:ceph: ceph version 16.2.14-456-g445562ab (445562ab4bc3ddfb386936119050695810860bcb) pacific (stable) 2024-01-13T08:56:38.988 INFO:teuthology.orchestra.run.smithi077.stderr:Ceph version: ceph version 16.2.14-456-g445562ab (445562ab4bc3ddfb386936119050695810860bcb) pacific (stable) 2024-01-13T08:56:38.988 INFO:teuthology.orchestra.run.smithi077.stderr:Extracting ceph user uid/gid from container image... 2024-01-13T08:56:39.268 INFO:teuthology.orchestra.run.smithi077.stderr:stat: 167 167 2024-01-13T08:56:39.590 INFO:teuthology.orchestra.run.smithi077.stderr:Creating initial keys... 2024-01-13T08:56:39.857 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-authtool: AQBHUKJl3g6KMhAAqjdA0bUrjGCqzGbpkaChqA== 2024-01-13T08:56:40.461 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-authtool: AQBIUKJlND44GRAAZiKprEnX6TjqelCnETkP9w== 2024-01-13T08:56:41.007 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-authtool: AQBJUKJlD1lYABAAwRqorGcxKs8haNlKTK6JzQ== 2024-01-13T08:56:41.340 INFO:teuthology.orchestra.run.smithi077.stderr:Creating initial monmap... 2024-01-13T08:56:41.574 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-13T08:56:41.574 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 99e41a38-b1f1-11ee-95ac-87774f69a715 2024-01-13T08:56:41.574 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-13T08:56:41.915 INFO:teuthology.orchestra.run.smithi077.stderr:monmaptool for smithi077 [v2:172.21.15.77:3300,v1:172.21.15.77:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-13T08:56:41.915 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/monmaptool: set fsid to 99e41a38-b1f1-11ee-95ac-87774f69a715 2024-01-13T08:56:41.915 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-13T08:56:41.915 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-01-13T08:56:41.916 INFO:teuthology.orchestra.run.smithi077.stderr:Creating mon... 2024-01-13T08:56:42.191 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-13T08:56:42.192 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 1 imported monmap: 2024-01-13T08:56:42.192 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: epoch 0 2024-01-13T08:56:42.192 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 2024-01-13T08:56:42.192 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: last_changed 2024-01-13T08:56:41.569521+0000 2024-01-13T08:56:42.192 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: created 2024-01-13T08:56:41.569521+0000 2024-01-13T08:56:42.192 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-01-13T08:56:42.192 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-01-13T08:56:42.192 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.smithi077 2024-01-13T08:56:42.192 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: 2024-01-13T08:56:42.192 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 0 /usr/bin/ceph-mon: set fsid to 99e41a38-b1f1-11ee-95ac-87774f69a715 2024-01-13T08:56:42.192 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: RocksDB version: 6.8.1 2024-01-13T08:56:42.193 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: 2024-01-13T08:56:42.193 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-13T08:56:42.193 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Compile date Jan 12 2024 2024-01-13T08:56:42.193 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: DB SUMMARY 2024-01-13T08:56:42.193 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: 2024-01-13T08:56:42.193 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi077/store.db dir, Total Num: 0, files: 2024-01-13T08:56:42.193 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: 2024-01-13T08:56:42.193 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi077/store.db: 2024-01-13T08:56:42.193 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: 2024-01-13T08:56:42.193 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.error_if_exists: 0 2024-01-13T08:56:42.193 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.create_if_missing: 1 2024-01-13T08:56:42.193 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.paranoid_checks: 1 2024-01-13T08:56:42.193 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.env: 0x55afe3a98080 2024-01-13T08:56:42.193 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.fs: Posix File System 2024-01-13T08:56:42.193 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.info_log: 0x55afe48d3380 2024-01-13T08:56:42.194 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-13T08:56:42.194 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.statistics: (nil) 2024-01-13T08:56:42.194 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.use_fsync: 0 2024-01-13T08:56:42.194 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.max_log_file_size: 0 2024-01-13T08:56:42.194 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-13T08:56:42.194 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-13T08:56:42.194 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-13T08:56:42.194 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-13T08:56:42.194 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.allow_fallocate: 1 2024-01-13T08:56:42.194 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-13T08:56:42.194 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-13T08:56:42.194 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.use_direct_reads: 0 2024-01-13T08:56:42.194 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-13T08:56:42.194 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.create_missing_column_families: 0 2024-01-13T08:56:42.194 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.db_log_dir: 2024-01-13T08:56:42.195 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi077/store.db 2024-01-13T08:56:42.195 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-13T08:56:42.195 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.max_subcompactions: 1 2024-01-13T08:56:42.195 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.max_background_flushes: -1 2024-01-13T08:56:42.195 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-13T08:56:42.195 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-13T08:56:42.195 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-13T08:56:42.195 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-13T08:56:42.195 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-13T08:56:42.195 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.advise_random_on_open: 1 2024-01-13T08:56:42.195 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-13T08:56:42.195 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.write_buffer_manager: 0x55afe48d9980 2024-01-13T08:56:42.195 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-13T08:56:42.195 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-13T08:56:42.195 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-13T08:56:42.195 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-13T08:56:42.196 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.rate_limiter: (nil) 2024-01-13T08:56:42.196 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-13T08:56:42.196 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-13T08:56:42.196 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-13T08:56:42.196 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-13T08:56:42.196 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.unordered_write: 0 2024-01-13T08:56:42.196 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-13T08:56:42.196 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-13T08:56:42.196 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-13T08:56:42.196 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-13T08:56:42.196 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.row_cache: None 2024-01-13T08:56:42.196 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.wal_filter: None 2024-01-13T08:56:42.196 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-13T08:56:42.196 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-13T08:56:42.197 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.preserve_deletes: 0 2024-01-13T08:56:42.197 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.two_write_queues: 0 2024-01-13T08:56:42.197 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.manual_wal_flush: 0 2024-01-13T08:56:42.197 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.atomic_flush: 0 2024-01-13T08:56:42.197 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-13T08:56:42.197 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-13T08:56:42.197 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-13T08:56:42.197 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.log_readahead_size: 0 2024-01-13T08:56:42.197 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-13T08:56:42.197 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.max_background_jobs: 2 2024-01-13T08:56:42.197 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.max_background_compactions: -1 2024-01-13T08:56:42.197 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-13T08:56:42.197 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-13T08:56:42.197 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-13T08:56:42.197 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.max_total_wal_size: 0 2024-01-13T08:56:42.197 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-13T08:56:42.198 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-13T08:56:42.198 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-13T08:56:42.198 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-13T08:56:42.198 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.max_open_files: -1 2024-01-13T08:56:42.198 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.bytes_per_sync: 0 2024-01-13T08:56:42.198 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-13T08:56:42.198 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-13T08:56:42.198 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-13T08:56:42.198 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Compression algorithms supported: 2024-01-13T08:56:42.198 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-13T08:56:42.198 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: kZSTD supported: 0 2024-01-13T08:56:42.199 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: kXpressCompression supported: 0 2024-01-13T08:56:42.199 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-13T08:56:42.199 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: kLZ4Compression supported: 1 2024-01-13T08:56:42.199 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: kBZip2Compression supported: 0 2024-01-13T08:56:42.199 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: kZlibCompression supported: 1 2024-01-13T08:56:42.199 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: kSnappyCompression supported: 1 2024-01-13T08:56:42.199 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.164+0000 7fa6b7e05880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-13T08:56:42.199 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-01-13T08:56:42.199 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: 2024-01-13T08:56:42.199 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi077/store.db/MANIFEST-000001 2024-01-13T08:56:42.199 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: 2024-01-13T08:56:42.199 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-13T08:56:42.199 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: 2024-01-13T08:56:42.199 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-13T08:56:42.200 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.merge_operator: 2024-01-13T08:56:42.200 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.compaction_filter: None 2024-01-13T08:56:42.200 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.compaction_filter_factory: None 2024-01-13T08:56:42.200 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-13T08:56:42.200 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-13T08:56:42.200 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55afe47e9d30) 2024-01-13T08:56:42.200 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-01-13T08:56:42.200 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-13T08:56:42.200 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-13T08:56:42.200 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-01-13T08:56:42.200 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: index_type: 0 2024-01-13T08:56:42.200 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-01-13T08:56:42.200 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-01-13T08:56:42.201 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-01-13T08:56:42.201 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-01-13T08:56:42.201 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: checksum: 1 2024-01-13T08:56:42.201 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-01-13T08:56:42.201 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: block_cache: 0x55afe481ed10 2024-01-13T08:56:42.201 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-01-13T08:56:42.201 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-01-13T08:56:42.201 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-01-13T08:56:42.201 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-01-13T08:56:42.201 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-01-13T08:56:42.201 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-01-13T08:56:42.201 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-01-13T08:56:42.201 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-01-13T08:56:42.201 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-01-13T08:56:42.202 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-01-13T08:56:42.202 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-01-13T08:56:42.202 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-01-13T08:56:42.202 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-01-13T08:56:42.202 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-01-13T08:56:42.202 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-01-13T08:56:42.202 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-13T08:56:42.202 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-01-13T08:56:42.202 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-01-13T08:56:42.202 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-01-13T08:56:42.202 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: format_version: 2 2024-01-13T08:56:42.202 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-01-13T08:56:42.202 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: block_align: 0 2024-01-13T08:56:42.202 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: 2024-01-13T08:56:42.203 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-13T08:56:42.203 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-13T08:56:42.203 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.compression: NoCompression 2024-01-13T08:56:42.203 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-13T08:56:42.203 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-13T08:56:42.203 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-13T08:56:42.203 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.num_levels: 7 2024-01-13T08:56:42.203 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-13T08:56:42.203 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-13T08:56:42.203 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-13T08:56:42.203 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-13T08:56:42.203 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-13T08:56:42.203 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-13T08:56:42.203 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-13T08:56:42.203 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-13T08:56:42.203 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-13T08:56:42.204 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-13T08:56:42.204 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.compression_opts.level: 32767 2024-01-13T08:56:42.204 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-13T08:56:42.204 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-13T08:56:42.204 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-13T08:56:42.204 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.compression_opts.enabled: false 2024-01-13T08:56:42.204 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-13T08:56:42.204 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-13T08:56:42.204 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-13T08:56:42.204 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-13T08:56:42.204 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-13T08:56:42.204 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-13T08:56:42.204 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-13T08:56:42.204 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-13T08:56:42.205 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-13T08:56:42.205 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-13T08:56:42.205 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-13T08:56:42.205 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-13T08:56:42.205 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-13T08:56:42.205 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-13T08:56:42.205 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-13T08:56:42.205 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-13T08:56:42.205 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-13T08:56:42.205 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.arena_block_size: 4194304 2024-01-13T08:56:42.205 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-13T08:56:42.205 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-13T08:56:42.205 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-13T08:56:42.205 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-13T08:56:42.206 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-13T08:56:42.206 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-13T08:56:42.206 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-13T08:56:42.206 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-13T08:56:42.206 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-13T08:56:42.206 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-13T08:56:42.206 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-13T08:56:42.206 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-13T08:56:42.206 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-13T08:56:42.206 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-13T08:56:42.206 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.table_properties_collectors: 2024-01-13T08:56:42.206 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.inplace_update_support: 0 2024-01-13T08:56:42.206 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-13T08:56:42.206 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-13T08:56:42.207 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-13T08:56:42.207 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-13T08:56:42.207 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.bloom_locality: 0 2024-01-13T08:56:42.207 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.max_successive_merges: 0 2024-01-13T08:56:42.207 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-13T08:56:42.207 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-13T08:56:42.207 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.force_consistency_checks: 0 2024-01-13T08:56:42.207 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-13T08:56:42.207 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.ttl: 2592000 2024-01-13T08:56:42.207 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.165+0000 7fa6b7e05880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-13T08:56:42.207 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.166+0000 7fa6b7e05880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi077/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-13T08:56:42.207 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: 2024-01-13T08:56:42.207 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.166+0000 7fa6b7e05880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-01-13T08:56:42.207 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: 2024-01-13T08:56:42.208 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.166+0000 7fa6b7e05880 4 rocksdb: DB pointer 0x55afe48e7800 2024-01-13T08:56:42.208 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.167+0000 7fa6a0c7a700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-13T08:56:42.208 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.167+0000 7fa6a0c7a700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-13T08:56:42.208 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-01-13T08:56:42.208 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T08:56:42.208 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-13T08:56:42.208 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-13T08:56:42.208 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-13T08:56:42.208 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-13T08:56:42.208 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-13T08:56:42.208 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-13T08:56:42.208 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: 2024-01-13T08:56:42.208 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-13T08:56:42.208 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T08:56:42.209 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T08:56:42.209 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-13T08:56:42.209 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-13T08:56:42.209 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: 2024-01-13T08:56:42.209 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-13T08:56:42.209 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T08:56:42.209 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T08:56:42.209 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T08:56:42.209 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-01-13T08:56:42.209 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-13T08:56:42.209 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-01-13T08:56:42.209 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-01-13T08:56:42.209 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-01-13T08:56:42.209 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T08:56:42.210 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T08:56:42.210 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-13T08:56:42.210 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: 2024-01-13T08:56:42.210 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-01-13T08:56:42.210 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: 2024-01-13T08:56:42.210 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-13T08:56:42.210 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T08:56:42.210 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T08:56:42.210 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-13T08:56:42.210 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-13T08:56:42.210 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: 2024-01-13T08:56:42.210 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-13T08:56:42.210 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T08:56:42.210 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T08:56:42.211 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T08:56:42.211 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-01-13T08:56:42.211 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-13T08:56:42.211 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-01-13T08:56:42.211 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-01-13T08:56:42.211 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-01-13T08:56:42.211 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T08:56:42.211 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T08:56:42.211 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-13T08:56:42.211 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: 2024-01-13T08:56:42.211 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-01-13T08:56:42.211 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: 2024-01-13T08:56:42.211 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.168+0000 7fa6b7e05880 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-01-13T08:56:42.211 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.168+0000 7fa6b7e05880 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-01-13T08:56:42.211 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:56:42.168+0000 7fa6b7e05880 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi077 for mon.smithi077 2024-01-13T08:56:42.546 INFO:teuthology.orchestra.run.smithi077.stderr:create mon.smithi077 on 2024-01-13T08:56:42.652 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-01-13T08:56:42.745 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-99e41a38-b1f1-11ee-95ac-87774f69a715.target → /etc/systemd/system/ceph-99e41a38-b1f1-11ee-95ac-87774f69a715.target. 2024-01-13T08:56:42.745 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-99e41a38-b1f1-11ee-95ac-87774f69a715.target → /etc/systemd/system/ceph-99e41a38-b1f1-11ee-95ac-87774f69a715.target. 2024-01-13T08:56:42.949 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: Failed to reset failed state of unit ceph-99e41a38-b1f1-11ee-95ac-87774f69a715@mon.smithi077.service: Unit ceph-99e41a38-b1f1-11ee-95ac-87774f69a715@mon.smithi077.service not loaded. 2024-01-13T08:56:42.956 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: Created symlink /etc/systemd/system/ceph-99e41a38-b1f1-11ee-95ac-87774f69a715.target.wants/ceph-99e41a38-b1f1-11ee-95ac-87774f69a715@mon.smithi077.service → /etc/systemd/system/ceph-99e41a38-b1f1-11ee-95ac-87774f69a715@.service. 2024-01-13T08:56:43.429 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: disabled 2024-01-13T08:56:43.436 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: inactive 2024-01-13T08:56:43.436 INFO:teuthology.orchestra.run.smithi077.stderr:firewalld.service is not enabled 2024-01-13T08:56:43.437 INFO:teuthology.orchestra.run.smithi077.stderr:Not possible to enable service . firewalld.service is not available 2024-01-13T08:56:43.437 INFO:teuthology.orchestra.run.smithi077.stderr:Waiting for mon to start... 2024-01-13T08:56:43.437 INFO:teuthology.orchestra.run.smithi077.stderr:Waiting for mon... 2024-01-13T08:56:43.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:43 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30363]: cluster 2024-01-13T08:56:43.429832+0000 mon.smithi077 (mon.0) 1 : cluster [INF] mon.smithi077 is new leader, mons smithi077 in quorum (ranks 0) 2024-01-13T08:56:43.911 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: cluster: 2024-01-13T08:56:43.911 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: id: 99e41a38-b1f1-11ee-95ac-87774f69a715 2024-01-13T08:56:43.911 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-01-13T08:56:43.911 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-01-13T08:56:43.911 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: services: 2024-01-13T08:56:43.911 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi077 (age 0.466979s) 2024-01-13T08:56:43.912 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: mgr: no daemons active 2024-01-13T08:56:43.912 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-01-13T08:56:43.912 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-01-13T08:56:43.912 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: data: 2024-01-13T08:56:43.912 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-01-13T08:56:43.912 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-01-13T08:56:43.912 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-01-13T08:56:43.912 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: pgs: 2024-01-13T08:56:43.912 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-01-13T08:56:44.260 INFO:teuthology.orchestra.run.smithi077.stderr:mon is available 2024-01-13T08:56:44.260 INFO:teuthology.orchestra.run.smithi077.stderr:Assimilating anything we can from ceph.conf... 2024-01-13T08:56:44.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:44 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30363]: cluster 2024-01-13T08:56:43.449739+0000 mon.smithi077 (mon.0) 2 : cluster [INF] mon.smithi077 is new leader, mons smithi077 in quorum (ranks 0) 2024-01-13T08:56:44.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:44 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30363]: cluster 2024-01-13T08:56:43.449979+0000 mon.smithi077 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi077=[v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0]} removed_ranks: {} 2024-01-13T08:56:44.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:44 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30363]: cluster 2024-01-13T08:56:43.465404+0000 mon.smithi077 (mon.0) 4 : cluster [DBG] fsmap 2024-01-13T08:56:44.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:44 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30363]: cluster 2024-01-13T08:56:43.468638+0000 mon.smithi077 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-01-13T08:56:44.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:44 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30363]: cluster 2024-01-13T08:56:43.469068+0000 mon.smithi077 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-01-13T08:56:44.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:44 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30363]: audit 2024-01-13T08:56:43.916655+0000 mon.smithi077 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.77:0/1445165771' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-01-13T08:56:44.686 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-01-13T08:56:44.686 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: [global] 2024-01-13T08:56:44.686 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: fsid = 99e41a38-b1f1-11ee-95ac-87774f69a715 2024-01-13T08:56:44.686 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.77:3300,v1:172.21.15.77:6789] 2024-01-13T08:56:44.686 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-01-13T08:56:44.686 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-01-13T08:56:44.687 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-01-13T08:56:44.687 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-01-13T08:56:44.687 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-01-13T08:56:44.687 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: [mgr] 2024-01-13T08:56:44.687 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-01-13T08:56:44.687 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-01-13T08:56:44.687 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: [osd] 2024-01-13T08:56:44.687 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-01-13T08:56:44.687 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-01-13T08:56:45.042 INFO:teuthology.orchestra.run.smithi077.stderr:Generating new minimal ceph.conf... 2024-01-13T08:56:45.761 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:45 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30363]: audit 2024-01-13T08:56:44.689816+0000 mon.smithi077 (mon.0) 8 : audit [INF] from='client.? 172.21.15.77:0/1071916681' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-13T08:56:45.761 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:45 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30363]: audit 2024-01-13T08:56:44.692308+0000 mon.smithi077 (mon.0) 9 : audit [INF] from='client.? 172.21.15.77:0/1071916681' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-01-13T08:56:45.762 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:45 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30363]: audit 2024-01-13T08:56:45.455413+0000 mon.smithi077 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.77:0/4091993223' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:56:45.793 INFO:teuthology.orchestra.run.smithi077.stderr:Restarting the monitor... 2024-01-13T08:56:46.020 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:45 smithi077 systemd[1]: Stopping Ceph mon.smithi077 for 99e41a38-b1f1-11ee-95ac-87774f69a715... 2024-01-13T08:56:46.021 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:45 smithi077 bash[30683]: Error: no container with name or ID "ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon.smithi077" found: no such container 2024-01-13T08:56:46.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30363]: debug 2024-01-13T08:56:46.028+0000 7f81f1431700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi077 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-13T08:56:46.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30363]: debug 2024-01-13T08:56:46.028+0000 7f81f1431700 -1 mon.smithi077@0(leader) e1 *** Got Signal Terminated *** 2024-01-13T08:56:46.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 bash[30722]: ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077 2024-01-13T08:56:46.852 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 systemd[1]: ceph-99e41a38-b1f1-11ee-95ac-87774f69a715@mon.smithi077.service: Succeeded. 2024-01-13T08:56:46.852 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 systemd[1]: Stopped Ceph mon.smithi077 for 99e41a38-b1f1-11ee-95ac-87774f69a715. 2024-01-13T08:56:46.852 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 systemd[1]: Starting Ceph mon.smithi077 for 99e41a38-b1f1-11ee-95ac-87774f69a715... 2024-01-13T08:56:46.894 INFO:teuthology.orchestra.run.smithi077.stderr:Setting public_network to 172.21.0.0/20 in mon config section 2024-01-13T08:56:47.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.896+0000 7f57c144c880 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-13T08:56:47.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.896+0000 7f57c144c880 0 ceph version 16.2.14-456-g445562ab (445562ab4bc3ddfb386936119050695810860bcb) pacific (stable), process ceph-mon, pid 7 2024-01-13T08:56:47.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 bash[30841]: cedb93a2d28f13a0b87e39032f9b5a6ea5f1076d1f787fd3fa085d7bedfd7634 2024-01-13T08:56:47.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.896+0000 7f57c144c880 0 pidfile_write: ignore empty --pid-file 2024-01-13T08:56:47.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 systemd[1]: Started Ceph mon.smithi077 for 99e41a38-b1f1-11ee-95ac-87774f69a715. 2024-01-13T08:56:47.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.901+0000 7f57c144c880 0 load: jerasure load: lrc load: isa 2024-01-13T08:56:47.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: RocksDB version: 6.8.1 2024-01-13T08:56:47.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: 2024-01-13T08:56:47.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-13T08:56:47.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Compile date Jan 12 2024 2024-01-13T08:56:47.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: DB SUMMARY 2024-01-13T08:56:47.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: 2024-01-13T08:56:47.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: CURRENT file: CURRENT 2024-01-13T08:56:47.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: 2024-01-13T08:56:47.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: IDENTITY file: IDENTITY 2024-01-13T08:56:47.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: 2024-01-13T08:56:47.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-01-13T08:56:47.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: 2024-01-13T08:56:47.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi077/store.db dir, Total Num: 1, files: 000004.sst 2024-01-13T08:56:47.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: 2024-01-13T08:56:47.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi077/store.db: 000006.log size: 81724 ; 2024-01-13T08:56:47.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: 2024-01-13T08:56:47.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.error_if_exists: 0 2024-01-13T08:56:47.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.create_if_missing: 0 2024-01-13T08:56:47.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.paranoid_checks: 1 2024-01-13T08:56:47.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.env: 0x55b521378080 2024-01-13T08:56:47.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.fs: Posix File System 2024-01-13T08:56:47.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.info_log: 0x55b523401d20 2024-01-13T08:56:47.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-13T08:56:47.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.statistics: (nil) 2024-01-13T08:56:47.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.use_fsync: 0 2024-01-13T08:56:47.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.max_log_file_size: 0 2024-01-13T08:56:47.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-13T08:56:47.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-13T08:56:47.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-13T08:56:47.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-13T08:56:47.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.allow_fallocate: 1 2024-01-13T08:56:47.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-13T08:56:47.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-13T08:56:47.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.use_direct_reads: 0 2024-01-13T08:56:47.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-13T08:56:47.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.create_missing_column_families: 0 2024-01-13T08:56:47.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.db_log_dir: 2024-01-13T08:56:47.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi077/store.db 2024-01-13T08:56:47.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-13T08:56:47.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.max_subcompactions: 1 2024-01-13T08:56:47.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.max_background_flushes: -1 2024-01-13T08:56:47.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-13T08:56:47.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-13T08:56:47.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-13T08:56:47.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-13T08:56:47.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-13T08:56:47.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.advise_random_on_open: 1 2024-01-13T08:56:47.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-13T08:56:47.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.write_buffer_manager: 0x55b524058480 2024-01-13T08:56:47.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-13T08:56:47.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-13T08:56:47.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-13T08:56:47.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-13T08:56:47.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.rate_limiter: (nil) 2024-01-13T08:56:47.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-13T08:56:47.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-13T08:56:47.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-13T08:56:47.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-13T08:56:47.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.unordered_write: 0 2024-01-13T08:56:47.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-13T08:56:47.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-13T08:56:47.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-13T08:56:47.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-13T08:56:47.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.row_cache: None 2024-01-13T08:56:47.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.wal_filter: None 2024-01-13T08:56:47.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-13T08:56:47.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-13T08:56:47.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.preserve_deletes: 0 2024-01-13T08:56:47.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.two_write_queues: 0 2024-01-13T08:56:47.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.manual_wal_flush: 0 2024-01-13T08:56:47.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.atomic_flush: 0 2024-01-13T08:56:47.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-13T08:56:47.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-13T08:56:47.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-13T08:56:47.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.log_readahead_size: 0 2024-01-13T08:56:47.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-13T08:56:47.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.max_background_jobs: 2 2024-01-13T08:56:47.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.max_background_compactions: -1 2024-01-13T08:56:47.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-13T08:56:47.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-13T08:56:47.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-13T08:56:47.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.max_total_wal_size: 0 2024-01-13T08:56:47.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-13T08:56:47.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-13T08:56:47.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-13T08:56:47.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-13T08:56:47.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.max_open_files: -1 2024-01-13T08:56:47.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.bytes_per_sync: 0 2024-01-13T08:56:47.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-13T08:56:47.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-13T08:56:47.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-13T08:56:47.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Compression algorithms supported: 2024-01-13T08:56:47.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-13T08:56:47.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: kZSTD supported: 0 2024-01-13T08:56:47.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: kXpressCompression supported: 0 2024-01-13T08:56:47.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-13T08:56:47.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: kLZ4Compression supported: 1 2024-01-13T08:56:47.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: kBZip2Compression supported: 0 2024-01-13T08:56:47.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: kZlibCompression supported: 1 2024-01-13T08:56:47.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: kSnappyCompression supported: 1 2024-01-13T08:56:47.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-13T08:56:47.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi077/store.db/MANIFEST-000005 2024-01-13T08:56:47.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: 2024-01-13T08:56:47.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-13T08:56:47.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: 2024-01-13T08:56:47.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-13T08:56:47.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.merge_operator: 2024-01-13T08:56:47.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.compaction_filter: None 2024-01-13T08:56:47.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.compaction_filter_factory: None 2024-01-13T08:56:47.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-13T08:56:47.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-13T08:56:47.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b5233b1d28) 2024-01-13T08:56:47.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cache_index_and_filter_blocks: 1 2024-01-13T08:56:47.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-13T08:56:47.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-13T08:56:47.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: pin_top_level_index_and_filter: 1 2024-01-13T08:56:47.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: index_type: 0 2024-01-13T08:56:47.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: data_block_index_type: 0 2024-01-13T08:56:47.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: index_shortening: 1 2024-01-13T08:56:47.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: data_block_hash_table_util_ratio: 0.750000 2024-01-13T08:56:47.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: hash_index_allow_collision: 1 2024-01-13T08:56:47.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: checksum: 1 2024-01-13T08:56:47.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: no_block_cache: 0 2024-01-13T08:56:47.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: block_cache: 0x55b5233e6f10 2024-01-13T08:56:47.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: block_cache_name: BinnedLRUCache 2024-01-13T08:56:47.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: block_cache_options: 2024-01-13T08:56:47.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: capacity : 536870912 2024-01-13T08:56:47.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: num_shard_bits : 4 2024-01-13T08:56:47.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: strict_capacity_limit : 0 2024-01-13T08:56:47.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: high_pri_pool_ratio: 0.000 2024-01-13T08:56:47.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: block_cache_compressed: (nil) 2024-01-13T08:56:47.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: persistent_cache: (nil) 2024-01-13T08:56:47.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: block_size: 4096 2024-01-13T08:56:47.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: block_size_deviation: 10 2024-01-13T08:56:47.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: block_restart_interval: 16 2024-01-13T08:56:47.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: index_block_restart_interval: 1 2024-01-13T08:56:47.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: metadata_block_size: 4096 2024-01-13T08:56:47.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: partition_filters: 0 2024-01-13T08:56:47.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: use_delta_encoding: 1 2024-01-13T08:56:47.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-13T08:56:47.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: whole_key_filtering: 1 2024-01-13T08:56:47.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: verify_compression: 0 2024-01-13T08:56:47.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: read_amp_bytes_per_bit: 0 2024-01-13T08:56:47.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: format_version: 2 2024-01-13T08:56:47.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: enable_index_compression: 1 2024-01-13T08:56:47.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: block_align: 0 2024-01-13T08:56:47.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: 2024-01-13T08:56:47.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-13T08:56:47.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-13T08:56:47.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.compression: NoCompression 2024-01-13T08:56:47.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-13T08:56:47.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-13T08:56:47.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-13T08:56:47.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.num_levels: 7 2024-01-13T08:56:47.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-13T08:56:47.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-13T08:56:47.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-13T08:56:47.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-13T08:56:47.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-13T08:56:47.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-13T08:56:47.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-13T08:56:47.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-13T08:56:47.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-13T08:56:47.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-13T08:56:47.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.compression_opts.level: 32767 2024-01-13T08:56:47.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-13T08:56:47.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-13T08:56:47.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-13T08:56:47.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.compression_opts.enabled: false 2024-01-13T08:56:47.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-13T08:56:47.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-13T08:56:47.341 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-13T08:56:47.341 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-13T08:56:47.341 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-13T08:56:47.341 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-13T08:56:47.341 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-13T08:56:47.341 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-13T08:56:47.341 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-13T08:56:47.341 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-13T08:56:47.341 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-13T08:56:47.341 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-13T08:56:47.341 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-13T08:56:47.341 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-13T08:56:47.341 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-13T08:56:47.341 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-13T08:56:47.341 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-13T08:56:47.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.arena_block_size: 4194304 2024-01-13T08:56:47.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-13T08:56:47.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-13T08:56:47.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-13T08:56:47.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-13T08:56:47.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-13T08:56:47.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-13T08:56:47.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-13T08:56:47.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-13T08:56:47.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-13T08:56:47.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-13T08:56:47.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-13T08:56:47.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-13T08:56:47.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-13T08:56:47.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-13T08:56:47.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.table_properties_collectors: 2024-01-13T08:56:47.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.inplace_update_support: 0 2024-01-13T08:56:47.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-13T08:56:47.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-13T08:56:47.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-13T08:56:47.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-13T08:56:47.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.bloom_locality: 0 2024-01-13T08:56:47.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.max_successive_merges: 0 2024-01-13T08:56:47.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-13T08:56:47.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-13T08:56:47.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.force_consistency_checks: 0 2024-01-13T08:56:47.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-13T08:56:47.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.ttl: 2592000 2024-01-13T08:56:47.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.902+0000 7f57c144c880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-13T08:56:47.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.903+0000 7f57c144c880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi077/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-13T08:56:47.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: 2024-01-13T08:56:47.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.903+0000 7f57c144c880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-01-13T08:56:47.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: 2024-01-13T08:56:47.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.903+0000 7f57c144c880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705136206903476, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-01-13T08:56:47.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.903+0000 7f57c144c880 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-01-13T08:56:47.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.903+0000 7f57c144c880 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-01-13T08:56:47.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.904+0000 7f57c144c880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705136206904341, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 78727, "table_properties": {"data_size": 77021, "index_size": 311, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 581, "raw_key_size": 9283, "raw_average_key_size": 47, "raw_value_size": 71591, "raw_average_value_size": 367, "num_data_blocks": 10, "num_entries": 195, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1705136206, "oldest_key_time": 3, "file_creation_time": 0}} 2024-01-13T08:56:47.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.904+0000 7f57c144c880 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-01-13T08:56:47.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: 2024-01-13T08:56:47.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.904+0000 7f57c144c880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705136206904759, "job": 1, "event": "recovery_finished"} 2024-01-13T08:56:47.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.905+0000 7f57c144c880 4 rocksdb: DB pointer 0x55b5234af800 2024-01-13T08:56:47.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.905+0000 7f57a8240700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-13T08:56:47.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.905+0000 7f57a8240700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-13T08:56:47.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: ** DB Stats ** 2024-01-13T08:56:47.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T08:56:47.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-13T08:56:47.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-13T08:56:47.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-13T08:56:47.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-13T08:56:47.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-13T08:56:47.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-13T08:56:47.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: 2024-01-13T08:56:47.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: ** Compaction Stats [default] ** 2024-01-13T08:56:47.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T08:56:47.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T08:56:47.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: L0 2/0 78.57 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 127.3 0.00 0.00 1 0.001 0 0 2024-01-13T08:56:47.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Sum 2/0 78.57 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 127.3 0.00 0.00 1 0.001 0 0 2024-01-13T08:56:47.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 127.3 0.00 0.00 1 0.001 0 0 2024-01-13T08:56:47.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: 2024-01-13T08:56:47.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: ** Compaction Stats [default] ** 2024-01-13T08:56:47.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T08:56:47.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T08:56:47.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 127.3 0.00 0.00 1 0.001 0 0 2024-01-13T08:56:47.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T08:56:47.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-13T08:56:47.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-13T08:56:47.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: AddFile(Total Files): cumulative 0, interval 0 2024-01-13T08:56:47.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-13T08:56:47.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: AddFile(Keys): cumulative 0, interval 0 2024-01-13T08:56:47.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Cumulative compaction: 0.00 GB write, 21.86 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T08:56:47.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Interval compaction: 0.00 GB write, 21.86 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T08:56:47.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-13T08:56:47.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: 2024-01-13T08:56:47.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: ** File Read Latency Histogram By Level [default] ** 2024-01-13T08:56:47.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: 2024-01-13T08:56:47.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: ** Compaction Stats [default] ** 2024-01-13T08:56:47.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T08:56:47.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T08:56:47.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: L0 2/0 78.57 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 127.3 0.00 0.00 1 0.001 0 0 2024-01-13T08:56:47.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Sum 2/0 78.57 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 127.3 0.00 0.00 1 0.001 0 0 2024-01-13T08:56:47.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-13T08:56:47.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: 2024-01-13T08:56:47.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: ** Compaction Stats [default] ** 2024-01-13T08:56:47.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T08:56:47.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T08:56:47.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 127.3 0.00 0.00 1 0.001 0 0 2024-01-13T08:56:47.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T08:56:47.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-13T08:56:47.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-13T08:56:47.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: AddFile(Total Files): cumulative 0, interval 0 2024-01-13T08:56:47.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-13T08:56:47.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: AddFile(Keys): cumulative 0, interval 0 2024-01-13T08:56:47.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Cumulative compaction: 0.00 GB write, 21.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T08:56:47.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T08:56:47.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-13T08:56:47.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: 2024-01-13T08:56:47.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: ** File Read Latency Histogram By Level [default] ** 2024-01-13T08:56:47.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: 2024-01-13T08:56:47.349 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.906+0000 7f57c144c880 0 starting mon.smithi077 rank 0 at public addrs [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] at bind addrs [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi077 fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 2024-01-13T08:56:47.349 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.907+0000 7f57c144c880 1 mon.smithi077@-1(???) e1 preinit fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 2024-01-13T08:56:47.349 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.907+0000 7f57c144c880 0 mon.smithi077@-1(???).mds e1 new map 2024-01-13T08:56:47.349 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.907+0000 7f57c144c880 0 mon.smithi077@-1(???).mds e1 print_map 2024-01-13T08:56:47.349 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: e1 2024-01-13T08:56:47.349 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: enable_multiple, ever_enabled_multiple: 1,1 2024-01-13T08:56:47.349 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-13T08:56:47.349 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: legacy client fscid: -1 2024-01-13T08:56:47.349 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: 2024-01-13T08:56:47.349 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: No filesystems configured 2024-01-13T08:56:47.349 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: 2024-01-13T08:56:47.349 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.907+0000 7f57c144c880 0 mon.smithi077@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-01-13T08:56:47.349 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.907+0000 7f57c144c880 0 mon.smithi077@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-13T08:56:47.349 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.907+0000 7f57c144c880 0 mon.smithi077@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-13T08:56:47.350 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.907+0000 7f57c144c880 0 mon.smithi077@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-13T08:56:47.350 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:56:44.689816+0000 mon.smithi077 (mon.0) 8 : audit [INF] from='client.? 172.21.15.77:0/1071916681' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-13T08:56:47.350 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:56:44.692308+0000 mon.smithi077 (mon.0) 9 : audit [INF] from='client.? 172.21.15.77:0/1071916681' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-01-13T08:56:47.350 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:56:45.455413+0000 mon.smithi077 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.77:0/4091993223' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:56:47.350 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T08:56:46.907+0000 7f57c144c880 1 mon.smithi077@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-01-13T08:56:47.350 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:56:46.922917+0000 mon.smithi077 (mon.0) 1 : cluster [INF] mon.smithi077 is new leader, mons smithi077 in quorum (ranks 0) 2024-01-13T08:56:47.350 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:56:46.922975+0000 mon.smithi077 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi077=[v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0]} removed_ranks: {} 2024-01-13T08:56:47.350 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:56:46.934780+0000 mon.smithi077 (mon.0) 3 : cluster [DBG] fsmap 2024-01-13T08:56:47.350 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:56:46.934816+0000 mon.smithi077 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-01-13T08:56:47.350 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:56:46.935212+0000 mon.smithi077 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-01-13T08:56:47.701 INFO:teuthology.orchestra.run.smithi077.stderr:Wrote config to /etc/ceph/ceph.conf 2024-01-13T08:56:47.701 INFO:teuthology.orchestra.run.smithi077.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-01-13T08:56:47.701 INFO:teuthology.orchestra.run.smithi077.stderr:Creating mgr... 2024-01-13T08:56:47.702 INFO:teuthology.orchestra.run.smithi077.stderr:Verifying port 9283 ... 2024-01-13T08:56:47.835 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: Failed to reset failed state of unit ceph-99e41a38-b1f1-11ee-95ac-87774f69a715@mgr.smithi077.amvfyi.service: Unit ceph-99e41a38-b1f1-11ee-95ac-87774f69a715@mgr.smithi077.amvfyi.service not loaded. 2024-01-13T08:56:47.840 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: Created symlink /etc/systemd/system/ceph-99e41a38-b1f1-11ee-95ac-87774f69a715.target.wants/ceph-99e41a38-b1f1-11ee-95ac-87774f69a715@mgr.smithi077.amvfyi.service → /etc/systemd/system/ceph-99e41a38-b1f1-11ee-95ac-87774f69a715@.service. 2024-01-13T08:56:48.313 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: disabled 2024-01-13T08:56:48.321 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: inactive 2024-01-13T08:56:48.322 INFO:teuthology.orchestra.run.smithi077.stderr:firewalld.service is not enabled 2024-01-13T08:56:48.322 INFO:teuthology.orchestra.run.smithi077.stderr:Not possible to enable service . firewalld.service is not available 2024-01-13T08:56:48.331 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: disabled 2024-01-13T08:56:48.339 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: inactive 2024-01-13T08:56:48.340 INFO:teuthology.orchestra.run.smithi077.stderr:firewalld.service is not enabled 2024-01-13T08:56:48.340 INFO:teuthology.orchestra.run.smithi077.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-01-13T08:56:48.340 INFO:teuthology.orchestra.run.smithi077.stderr:Waiting for mgr to start... 2024-01-13T08:56:48.340 INFO:teuthology.orchestra.run.smithi077.stderr:Waiting for mgr... 2024-01-13T08:56:48.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:48 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:56:47.318284+0000 mon.smithi077 (mon.0) 6 : audit [INF] from='client.? 172.21.15.77:0/664939602' entity='client.admin' 2024-01-13T08:56:48.781 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-01-13T08:56:48.781 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: { 2024-01-13T08:56:48.781 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "fsid": "99e41a38-b1f1-11ee-95ac-87774f69a715", 2024-01-13T08:56:48.781 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "health": { 2024-01-13T08:56:48.782 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-13T08:56:48.782 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "checks": {}, 2024-01-13T08:56:48.782 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "mutes": [] 2024-01-13T08:56:48.782 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-01-13T08:56:48.782 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-13T08:56:48.782 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "quorum": [ 2024-01-13T08:56:48.782 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 0 2024-01-13T08:56:48.782 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: ], 2024-01-13T08:56:48.782 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-13T08:56:48.782 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "smithi077" 2024-01-13T08:56:48.782 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: ], 2024-01-13T08:56:48.783 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "quorum_age": 1, 2024-01-13T08:56:48.783 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "monmap": { 2024-01-13T08:56:48.783 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T08:56:48.783 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-13T08:56:48.783 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-13T08:56:48.783 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-01-13T08:56:48.783 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "osdmap": { 2024-01-13T08:56:48.783 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T08:56:48.783 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-13T08:56:48.783 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-13T08:56:48.783 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-13T08:56:48.787 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-13T08:56:48.787 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-13T08:56:48.788 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-13T08:56:48.788 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-01-13T08:56:48.788 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "pgmap": { 2024-01-13T08:56:48.788 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-13T08:56:48.788 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-13T08:56:48.788 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-13T08:56:48.788 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-13T08:56:48.788 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-13T08:56:48.788 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-13T08:56:48.788 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-13T08:56:48.789 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-13T08:56:48.789 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-01-13T08:56:48.789 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "fsmap": { 2024-01-13T08:56:48.789 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T08:56:48.789 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-13T08:56:48.789 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-13T08:56:48.789 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-01-13T08:56:48.789 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-13T08:56:48.789 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "available": false, 2024-01-13T08:56:48.789 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-13T08:56:48.789 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "modules": [ 2024-01-13T08:56:48.789 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "iostat", 2024-01-13T08:56:48.789 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "nfs", 2024-01-13T08:56:48.790 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "restful" 2024-01-13T08:56:48.790 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: ], 2024-01-13T08:56:48.790 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "services": {} 2024-01-13T08:56:48.790 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-01-13T08:56:48.790 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "servicemap": { 2024-01-13T08:56:48.790 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T08:56:48.790 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "modified": "2024-01-13T08:56:43.463616+0000", 2024-01-13T08:56:48.790 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "services": {} 2024-01-13T08:56:48.790 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-01-13T08:56:48.790 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-13T08:56:48.790 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: } 2024-01-13T08:56:49.079 INFO:teuthology.orchestra.run.smithi077.stderr:mgr not available, waiting (1/15)... 2024-01-13T08:56:49.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:49 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:56:48.784996+0000 mon.smithi077 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.77:0/1076082300' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-13T08:56:52.619 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-01-13T08:56:52.620 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: { 2024-01-13T08:56:52.620 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "fsid": "99e41a38-b1f1-11ee-95ac-87774f69a715", 2024-01-13T08:56:52.620 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "health": { 2024-01-13T08:56:52.620 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-13T08:56:52.620 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "checks": {}, 2024-01-13T08:56:52.620 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "mutes": [] 2024-01-13T08:56:52.620 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-01-13T08:56:52.620 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-13T08:56:52.620 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "quorum": [ 2024-01-13T08:56:52.620 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 0 2024-01-13T08:56:52.620 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: ], 2024-01-13T08:56:52.621 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-13T08:56:52.621 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "smithi077" 2024-01-13T08:56:52.621 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: ], 2024-01-13T08:56:52.621 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "quorum_age": 5, 2024-01-13T08:56:52.621 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "monmap": { 2024-01-13T08:56:52.621 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T08:56:52.621 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-13T08:56:52.621 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-13T08:56:52.621 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-01-13T08:56:52.621 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "osdmap": { 2024-01-13T08:56:52.621 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T08:56:52.621 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-13T08:56:52.622 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-13T08:56:52.622 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-13T08:56:52.622 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-13T08:56:52.622 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-13T08:56:52.622 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-13T08:56:52.622 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-01-13T08:56:52.622 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "pgmap": { 2024-01-13T08:56:52.622 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-13T08:56:52.622 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-13T08:56:52.622 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-13T08:56:52.622 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-13T08:56:52.622 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-13T08:56:52.622 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-13T08:56:52.623 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-13T08:56:52.623 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-13T08:56:52.623 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-01-13T08:56:52.623 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "fsmap": { 2024-01-13T08:56:52.623 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T08:56:52.623 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-13T08:56:52.623 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-13T08:56:52.623 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-01-13T08:56:52.623 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-13T08:56:52.623 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "available": false, 2024-01-13T08:56:52.623 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-13T08:56:52.623 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "modules": [ 2024-01-13T08:56:52.624 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "iostat", 2024-01-13T08:56:52.624 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "nfs", 2024-01-13T08:56:52.624 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "restful" 2024-01-13T08:56:52.624 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: ], 2024-01-13T08:56:52.624 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "services": {} 2024-01-13T08:56:52.624 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-01-13T08:56:52.624 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "servicemap": { 2024-01-13T08:56:52.624 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T08:56:52.624 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "modified": "2024-01-13T08:56:43.463616+0000", 2024-01-13T08:56:52.624 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "services": {} 2024-01-13T08:56:52.624 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-01-13T08:56:52.624 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-13T08:56:52.624 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: } 2024-01-13T08:56:53.098 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:56:52.603794+0000 mon.smithi077 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.77:0/2055262010' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-13T08:56:53.419 INFO:teuthology.orchestra.run.smithi077.stderr:mgr not available, waiting (2/15)... 2024-01-13T08:56:54.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:56:53.639051+0000 mon.smithi077 (mon.0) 9 : cluster [INF] Activating manager daemon smithi077.amvfyi 2024-01-13T08:56:54.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:56:53.641692+0000 mon.smithi077 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi077.amvfyi(active, starting, since 0.00271781s) 2024-01-13T08:56:54.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:56:53.642420+0000 mon.smithi077 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.77:0/337313961' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-13T08:56:54.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:56:53.642565+0000 mon.smithi077 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.77:0/337313961' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-13T08:56:54.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:56:53.642670+0000 mon.smithi077 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.77:0/337313961' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-13T08:56:54.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:56:53.642807+0000 mon.smithi077 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.77:0/337313961' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-01-13T08:56:54.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:56:53.643162+0000 mon.smithi077 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.77:0/337313961' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mgr metadata", "who": "smithi077.amvfyi", "id": "smithi077.amvfyi"}]: dispatch 2024-01-13T08:56:54.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:56:53.656611+0000 mon.smithi077 (mon.0) 16 : cluster [INF] Manager daemon smithi077.amvfyi is now available 2024-01-13T08:56:54.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:56:53.667026+0000 mon.smithi077 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.77:0/337313961' entity='mgr.smithi077.amvfyi' 2024-01-13T08:56:54.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:56:53.670590+0000 mon.smithi077 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.77:0/337313961' entity='mgr.smithi077.amvfyi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.amvfyi/mirror_snapshot_schedule"}]: dispatch 2024-01-13T08:56:54.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:56:53.673507+0000 mon.smithi077 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.77:0/337313961' entity='mgr.smithi077.amvfyi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.amvfyi/trash_purge_schedule"}]: dispatch 2024-01-13T08:56:54.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:56:53.678129+0000 mon.smithi077 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.77:0/337313961' entity='mgr.smithi077.amvfyi' 2024-01-13T08:56:54.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:56:53.683302+0000 mon.smithi077 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.77:0/337313961' entity='mgr.smithi077.amvfyi' 2024-01-13T08:56:55.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:55 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:56:54.644636+0000 mon.smithi077 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi077.amvfyi(active, since 1.00566s) 2024-01-13T08:56:56.011 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-01-13T08:56:56.011 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: { 2024-01-13T08:56:56.011 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "fsid": "99e41a38-b1f1-11ee-95ac-87774f69a715", 2024-01-13T08:56:56.011 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "health": { 2024-01-13T08:56:56.011 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-13T08:56:56.011 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "checks": {}, 2024-01-13T08:56:56.012 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "mutes": [] 2024-01-13T08:56:56.012 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-01-13T08:56:56.012 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-13T08:56:56.012 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "quorum": [ 2024-01-13T08:56:56.012 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 0 2024-01-13T08:56:56.012 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: ], 2024-01-13T08:56:56.012 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-13T08:56:56.012 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "smithi077" 2024-01-13T08:56:56.012 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: ], 2024-01-13T08:56:56.012 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "quorum_age": 9, 2024-01-13T08:56:56.012 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "monmap": { 2024-01-13T08:56:56.012 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T08:56:56.012 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-13T08:56:56.012 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-13T08:56:56.013 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-01-13T08:56:56.013 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "osdmap": { 2024-01-13T08:56:56.013 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T08:56:56.013 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-13T08:56:56.013 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-13T08:56:56.013 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-13T08:56:56.013 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-13T08:56:56.013 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-13T08:56:56.013 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-13T08:56:56.013 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-01-13T08:56:56.013 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "pgmap": { 2024-01-13T08:56:56.013 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-13T08:56:56.013 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-13T08:56:56.013 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-13T08:56:56.013 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-13T08:56:56.014 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-13T08:56:56.014 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-13T08:56:56.014 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-13T08:56:56.014 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-13T08:56:56.014 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-01-13T08:56:56.014 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "fsmap": { 2024-01-13T08:56:56.014 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T08:56:56.014 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-13T08:56:56.014 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-13T08:56:56.014 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-01-13T08:56:56.014 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-13T08:56:56.014 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "available": true, 2024-01-13T08:56:56.014 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-13T08:56:56.014 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "modules": [ 2024-01-13T08:56:56.014 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "iostat", 2024-01-13T08:56:56.015 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "nfs", 2024-01-13T08:56:56.015 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "restful" 2024-01-13T08:56:56.015 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: ], 2024-01-13T08:56:56.015 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "services": {} 2024-01-13T08:56:56.015 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-01-13T08:56:56.015 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "servicemap": { 2024-01-13T08:56:56.015 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T08:56:56.015 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "modified": "2024-01-13T08:56:43.463616+0000", 2024-01-13T08:56:56.015 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "services": {} 2024-01-13T08:56:56.015 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-01-13T08:56:56.015 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-13T08:56:56.015 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: } 2024-01-13T08:56:56.341 INFO:teuthology.orchestra.run.smithi077.stderr:mgr is available 2024-01-13T08:56:56.864 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-01-13T08:56:56.864 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: [global] 2024-01-13T08:56:56.864 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: fsid = 99e41a38-b1f1-11ee-95ac-87774f69a715 2024-01-13T08:56:56.864 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-01-13T08:56:56.864 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-01-13T08:56:56.864 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-01-13T08:56:56.864 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-01-13T08:56:56.864 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-01-13T08:56:56.865 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: [mgr] 2024-01-13T08:56:56.865 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-01-13T08:56:56.865 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-01-13T08:56:56.865 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: [osd] 2024-01-13T08:56:56.865 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-01-13T08:56:56.865 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-01-13T08:56:56.887 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:56 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:56:56.013112+0000 mon.smithi077 (mon.0) 23 : audit [DBG] from='client.? 172.21.15.77:0/1595116111' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-13T08:56:57.258 INFO:teuthology.orchestra.run.smithi077.stderr:Enabling cephadm module... 2024-01-13T08:56:58.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:57 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:56:56.644455+0000 mon.smithi077 (mon.0) 24 : cluster [DBG] mgrmap e4: smithi077.amvfyi(active, since 3s) 2024-01-13T08:56:58.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:57 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:56:56.869897+0000 mon.smithi077 (mon.0) 25 : audit [INF] from='client.? 172.21.15.77:0/3173215695' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-13T08:56:58.943 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:58 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:56:57.840159+0000 mon.smithi077 (mon.0) 26 : audit [INF] from='client.? 172.21.15.77:0/334216454' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-01-13T08:56:59.559 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: { 2024-01-13T08:56:59.559 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 5, 2024-01-13T08:56:59.559 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "available": true, 2024-01-13T08:56:59.559 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "active_name": "smithi077.amvfyi", 2024-01-13T08:56:59.559 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_standby": 0 2024-01-13T08:56:59.559 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: } 2024-01-13T08:56:59.900 INFO:teuthology.orchestra.run.smithi077.stderr:Waiting for the mgr to restart... 2024-01-13T08:56:59.900 INFO:teuthology.orchestra.run.smithi077.stderr:Waiting for mgr epoch 5... 2024-01-13T08:57:00.032 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:59 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:56:58.650755+0000 mon.smithi077 (mon.0) 27 : audit [INF] from='client.? 172.21.15.77:0/334216454' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-01-13T08:57:00.032 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:59 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:56:58.650803+0000 mon.smithi077 (mon.0) 28 : cluster [DBG] mgrmap e5: smithi077.amvfyi(active, since 5s) 2024-01-13T08:57:00.032 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:56:59 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:56:59.565165+0000 mon.smithi077 (mon.0) 29 : audit [DBG] from='client.? 172.21.15.77:0/3486543710' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-13T08:57:04.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:04 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:57:03.981187+0000 mon.smithi077 (mon.0) 30 : cluster [INF] Active manager daemon smithi077.amvfyi restarted 2024-01-13T08:57:04.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:04 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:57:03.981397+0000 mon.smithi077 (mon.0) 31 : cluster [INF] Activating manager daemon smithi077.amvfyi 2024-01-13T08:57:04.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:04 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:57:03.984061+0000 mon.smithi077 (mon.0) 32 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-01-13T08:57:04.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:04 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:57:03.984159+0000 mon.smithi077 (mon.0) 33 : cluster [DBG] mgrmap e6: smithi077.amvfyi(active, starting, since 0.00283703s) 2024-01-13T08:57:04.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:04 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:03.984934+0000 mon.smithi077 (mon.0) 34 : audit [DBG] from='mgr.14116 172.21.15.77:0/3732365360' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-01-13T08:57:04.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:04 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:03.985615+0000 mon.smithi077 (mon.0) 35 : audit [DBG] from='mgr.14116 172.21.15.77:0/3732365360' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mgr metadata", "who": "smithi077.amvfyi", "id": "smithi077.amvfyi"}]: dispatch 2024-01-13T08:57:04.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:04 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:03.986248+0000 mon.smithi077 (mon.0) 36 : audit [DBG] from='mgr.14116 172.21.15.77:0/3732365360' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-13T08:57:04.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:04 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:03.986406+0000 mon.smithi077 (mon.0) 37 : audit [DBG] from='mgr.14116 172.21.15.77:0/3732365360' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-13T08:57:04.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:04 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:03.986525+0000 mon.smithi077 (mon.0) 38 : audit [DBG] from='mgr.14116 172.21.15.77:0/3732365360' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-13T08:57:04.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:04 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:57:04.001558+0000 mon.smithi077 (mon.0) 39 : cluster [INF] Manager daemon smithi077.amvfyi is now available 2024-01-13T08:57:04.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:04 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:04.014385+0000 mon.smithi077 (mon.0) 40 : audit [INF] from='mgr.14116 172.21.15.77:0/3732365360' entity='mgr.smithi077.amvfyi' 2024-01-13T08:57:04.986 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: { 2024-01-13T08:57:04.986 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-01-13T08:57:04.987 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "initialized": true 2024-01-13T08:57:04.987 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: } 2024-01-13T08:57:05.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:05 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:57:04.011856+0000 mgr.smithi077.amvfyi (mgr.14116) 1 : cephadm [INF] Found migration_current of "None". Setting to last migration. 2024-01-13T08:57:05.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:05 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:04.100480+0000 mon.smithi077 (mon.0) 41 : audit [INF] from='mgr.14116 172.21.15.77:0/3732365360' entity='mgr.smithi077.amvfyi' 2024-01-13T08:57:05.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:05 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:04.101523+0000 mon.smithi077 (mon.0) 42 : audit [DBG] from='mgr.14116 172.21.15.77:0/3732365360' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:57:05.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:05 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:04.102769+0000 mon.smithi077 (mon.0) 43 : audit [DBG] from='mgr.14116 172.21.15.77:0/3732365360' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:57:05.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:05 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:04.104859+0000 mon.smithi077 (mon.0) 44 : audit [DBG] from='mgr.14116 172.21.15.77:0/3732365360' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:57:05.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:05 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:04.118066+0000 mon.smithi077 (mon.0) 45 : audit [INF] from='mgr.14116 172.21.15.77:0/3732365360' entity='mgr.smithi077.amvfyi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.amvfyi/mirror_snapshot_schedule"}]: dispatch 2024-01-13T08:57:05.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:05 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:04.120921+0000 mon.smithi077 (mon.0) 46 : audit [INF] from='mgr.14116 172.21.15.77:0/3732365360' entity='mgr.smithi077.amvfyi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.amvfyi/trash_purge_schedule"}]: dispatch 2024-01-13T08:57:05.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:05 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:57:04.987268+0000 mon.smithi077 (mon.0) 47 : cluster [DBG] mgrmap e7: smithi077.amvfyi(active, since 1.00594s) 2024-01-13T08:57:05.986 INFO:teuthology.orchestra.run.smithi077.stderr:mgr epoch 5 is available 2024-01-13T08:57:05.986 INFO:teuthology.orchestra.run.smithi077.stderr:Setting orchestrator backend to cephadm... 2024-01-13T08:57:06.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:06 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:04.986296+0000 mgr.smithi077.amvfyi (mgr.14116) 2 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-13T08:57:06.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:06 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:04.993320+0000 mgr.smithi077.amvfyi (mgr.14116) 3 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-13T08:57:07.452 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:07 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:57:06.104842+0000 mon.smithi077 (mon.0) 48 : cluster [DBG] mgrmap e8: smithi077.amvfyi(active, since 2s) 2024-01-13T08:57:08.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:08 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:07.428352+0000 mgr.smithi077.amvfyi (mgr.14116) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T08:57:08.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:08 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:07.432953+0000 mon.smithi077 (mon.0) 49 : audit [INF] from='mgr.14116 172.21.15.77:0/3732365360' entity='mgr.smithi077.amvfyi' 2024-01-13T08:57:08.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:08 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:07.481013+0000 mon.smithi077 (mon.0) 50 : audit [DBG] from='mgr.14116 172.21.15.77:0/3732365360' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:57:10.916 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: value unchanged 2024-01-13T08:57:11.265 INFO:teuthology.orchestra.run.smithi077.stderr:Generating ssh key... 2024-01-13T08:57:12.053 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:11 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:10.923888+0000 mgr.smithi077.amvfyi (mgr.14116) 5 : audit [DBG] from='client.14128 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T08:57:12.905 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtBLew986nMMKO+PX4BFHdncEo3FM0poW9t5aedrqMiAAzujcsX+aYPOFABULx8GRD0HFxEKy5yB54lkLksgDigE4re+mDRXl8ecYoMk6ol6+jH5cDiNR9U5O8V2PJaNIzFn0awNv4U81SaBVZcbIhVnMhhRMgm8uYW45KizJnffZ3ftP9wOSgPT6fnp9UPzOR7gvqmPat3DIEmUVPFmwHMsni/Bkv8aVwWVC80yUjzxuCO6gh65bNSntJxfP449FVlsOcMCxkHbvgDYJHc9YQ3uhZ11HFPm2gP5mCEHTOUTeNZyyTVVUAG8Q/boEUm/xi8tdF2MbEO2i7EJR7tjyX1ZLkPBysIGqs5Ie1nOcuSFQs05dC6Vh9gc4TFnasApc1Zc9JR/Ik82YC5eRHvD4vdAjSzZgvGtUiAEOXP8ADbfgxgdOJvnFRrJLmiGwqA1A9gfk/Whkie2UpoyJXX8EnyOvhrGd3fAqfbED+BrqPGLwmsYhlQ4Lc7JGKaIJKOgM= ceph-99e41a38-b1f1-11ee-95ac-87774f69a715 2024-01-13T08:57:13.256 INFO:teuthology.orchestra.run.smithi077.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-01-13T08:57:13.256 INFO:teuthology.orchestra.run.smithi077.stderr:Adding key to root@localhost authorized_keys... 2024-01-13T08:57:13.257 INFO:teuthology.orchestra.run.smithi077.stderr:Adding host smithi077... 2024-01-13T08:57:13.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:13 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:11.768590+0000 mgr.smithi077.amvfyi (mgr.14116) 6 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T08:57:13.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:13 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:57:11.768807+0000 mgr.smithi077.amvfyi (mgr.14116) 7 : cephadm [INF] Generating ssh key... 2024-01-13T08:57:13.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:13 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:12.063707+0000 mon.smithi077 (mon.0) 51 : audit [INF] from='mgr.14116 172.21.15.77:0/3732365360' entity='mgr.smithi077.amvfyi' 2024-01-13T08:57:13.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:13 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:12.065377+0000 mon.smithi077 (mon.0) 52 : audit [INF] from='mgr.14116 172.21.15.77:0/3732365360' entity='mgr.smithi077.amvfyi' 2024-01-13T08:57:14.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:12.911907+0000 mgr.smithi077.amvfyi (mgr.14116) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T08:57:14.370 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: Added host 'smithi077' with addr '172.21.15.77' 2024-01-13T08:57:14.701 INFO:teuthology.orchestra.run.smithi077.stderr:Deploying mon service with default placement... 2024-01-13T08:57:15.318 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: Scheduled mon update... 2024-01-13T08:57:15.570 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:13.767206+0000 mgr.smithi077.amvfyi (mgr.14116) 9 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi077", "addr": "172.21.15.77", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T08:57:15.570 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:14.376408+0000 mon.smithi077 (mon.0) 53 : audit [INF] from='mgr.14116 172.21.15.77:0/3732365360' entity='mgr.smithi077.amvfyi' 2024-01-13T08:57:15.570 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:57:14.376666+0000 mgr.smithi077.amvfyi (mgr.14116) 10 : cephadm [INF] Added host smithi077 2024-01-13T08:57:15.570 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:14.409870+0000 mon.smithi077 (mon.0) 54 : audit [DBG] from='mgr.14116 172.21.15.77:0/3732365360' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:57:15.570 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:15.324099+0000 mon.smithi077 (mon.0) 55 : audit [INF] from='mgr.14116 172.21.15.77:0/3732365360' entity='mgr.smithi077.amvfyi' 2024-01-13T08:57:15.713 INFO:teuthology.orchestra.run.smithi077.stderr:Deploying mgr service with default placement... 2024-01-13T08:57:16.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:16 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:15.320653+0000 mgr.smithi077.amvfyi (mgr.14116) 11 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T08:57:16.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:16 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:57:15.321829+0000 mgr.smithi077.amvfyi (mgr.14116) 12 : cephadm [INF] Saving service mon spec with placement count:5 2024-01-13T08:57:16.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:16 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:16.094753+0000 mon.smithi077 (mon.0) 56 : audit [INF] from='mgr.14116 172.21.15.77:0/3732365360' entity='mgr.smithi077.amvfyi' 2024-01-13T08:57:16.586 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-01-13T08:57:17.221 INFO:teuthology.orchestra.run.smithi077.stderr:Deploying crash service with default placement... 2024-01-13T08:57:17.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:17 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:16.391349+0000 mon.smithi077 (mon.0) 57 : audit [INF] from='mgr.14116 172.21.15.77:0/3732365360' entity='mgr.smithi077.amvfyi' 2024-01-13T08:57:17.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:17 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:16.589462+0000 mgr.smithi077.amvfyi (mgr.14116) 13 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T08:57:17.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:17 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:57:16.590287+0000 mgr.smithi077.amvfyi (mgr.14116) 14 : cephadm [INF] Saving service mgr spec with placement count:2 2024-01-13T08:57:17.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:17 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:16.593375+0000 mon.smithi077 (mon.0) 58 : audit [INF] from='mgr.14116 172.21.15.77:0/3732365360' entity='mgr.smithi077.amvfyi' 2024-01-13T08:57:17.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:17 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:17.108213+0000 mon.smithi077 (mon.0) 59 : audit [INF] from='mgr.14116 172.21.15.77:0/3732365360' entity='mgr.smithi077.amvfyi' 2024-01-13T08:57:17.790 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: Scheduled crash update... 2024-01-13T08:57:18.346 INFO:teuthology.orchestra.run.smithi077.stderr:Deploying prometheus service with default placement... 2024-01-13T08:57:18.855 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-01-13T08:57:19.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:18 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:17.793547+0000 mgr.smithi077.amvfyi (mgr.14116) 15 : audit [DBG] from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T08:57:19.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:18 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:57:17.794624+0000 mgr.smithi077.amvfyi (mgr.14116) 16 : cephadm [INF] Saving service crash spec with placement * 2024-01-13T08:57:19.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:18 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:17.796940+0000 mon.smithi077 (mon.0) 60 : audit [INF] from='mgr.14116 172.21.15.77:0/3732365360' entity='mgr.smithi077.amvfyi' 2024-01-13T08:57:19.214 INFO:teuthology.orchestra.run.smithi077.stderr:Deploying grafana service with default placement... 2024-01-13T08:57:19.718 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-01-13T08:57:20.015 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:19 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:18.859554+0000 mgr.smithi077.amvfyi (mgr.14116) 17 : audit [DBG] from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T08:57:20.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:19 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:57:18.860431+0000 mgr.smithi077.amvfyi (mgr.14116) 18 : cephadm [INF] Saving service prometheus spec with placement count:1 2024-01-13T08:57:20.059 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:19 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:18.862585+0000 mon.smithi077 (mon.0) 61 : audit [INF] from='mgr.14116 172.21.15.77:0/3732365360' entity='mgr.smithi077.amvfyi' 2024-01-13T08:57:20.059 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:19 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:19.725521+0000 mon.smithi077 (mon.0) 62 : audit [INF] from='mgr.14116 172.21.15.77:0/3732365360' entity='mgr.smithi077.amvfyi' 2024-01-13T08:57:20.063 INFO:teuthology.orchestra.run.smithi077.stderr:Deploying node-exporter service with default placement... 2024-01-13T08:57:20.560 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-01-13T08:57:20.897 INFO:teuthology.orchestra.run.smithi077.stderr:Deploying alertmanager service with default placement... 2024-01-13T08:57:21.392 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-01-13T08:57:21.690 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:19.722579+0000 mgr.smithi077.amvfyi (mgr.14116) 19 : audit [DBG] from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T08:57:21.690 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:57:19.723341+0000 mgr.smithi077.amvfyi (mgr.14116) 20 : cephadm [INF] Saving service grafana spec with placement count:1 2024-01-13T08:57:21.690 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:20.564341+0000 mgr.smithi077.amvfyi (mgr.14116) 21 : audit [DBG] from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T08:57:21.690 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:57:20.565140+0000 mgr.smithi077.amvfyi (mgr.14116) 22 : cephadm [INF] Saving service node-exporter spec with placement * 2024-01-13T08:57:21.691 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:20.567296+0000 mon.smithi077 (mon.0) 63 : audit [INF] from='mgr.14116 172.21.15.77:0/3732365360' entity='mgr.smithi077.amvfyi' 2024-01-13T08:57:21.691 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:21.399218+0000 mon.smithi077 (mon.0) 64 : audit [INF] from='mgr.14116 172.21.15.77:0/3732365360' entity='mgr.smithi077.amvfyi' 2024-01-13T08:57:22.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:21.396330+0000 mgr.smithi077.amvfyi (mgr.14116) 23 : audit [DBG] from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T08:57:22.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:57:21.397084+0000 mgr.smithi077.amvfyi (mgr.14116) 24 : cephadm [INF] Saving service alertmanager spec with placement count:1 2024-01-13T08:57:22.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:22.290201+0000 mon.smithi077 (mon.0) 65 : audit [INF] from='client.? 172.21.15.77:0/4016226849' entity='client.admin' 2024-01-13T08:57:23.454 INFO:teuthology.orchestra.run.smithi077.stderr:Enabling the dashboard module... 2024-01-13T08:57:24.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:24 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:23.130925+0000 mon.smithi077 (mon.0) 66 : audit [INF] from='client.? 172.21.15.77:0/3099617981' entity='client.admin' 2024-01-13T08:57:24.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:24 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:24.030225+0000 mon.smithi077 (mon.0) 67 : audit [INF] from='client.? 172.21.15.77:0/2086353071' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-01-13T08:57:25.282 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:25 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:57:23.987647+0000 mgr.smithi077.amvfyi (mgr.14116) 25 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:57:25.282 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:25 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:24.992264+0000 mon.smithi077 (mon.0) 68 : audit [INF] from='client.? 172.21.15.77:0/2086353071' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-01-13T08:57:25.282 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:25 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:57:24.992327+0000 mon.smithi077 (mon.0) 69 : cluster [DBG] mgrmap e9: smithi077.amvfyi(active, since 21s) 2024-01-13T08:57:25.907 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: { 2024-01-13T08:57:25.908 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 9, 2024-01-13T08:57:25.908 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "available": true, 2024-01-13T08:57:25.908 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "active_name": "smithi077.amvfyi", 2024-01-13T08:57:25.908 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_standby": 0 2024-01-13T08:57:25.908 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: } 2024-01-13T08:57:26.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:26 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:25.914414+0000 mon.smithi077 (mon.0) 70 : audit [DBG] from='client.? 172.21.15.77:0/2663121314' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-13T08:57:26.400 INFO:teuthology.orchestra.run.smithi077.stderr:Waiting for the mgr to restart... 2024-01-13T08:57:26.400 INFO:teuthology.orchestra.run.smithi077.stderr:Waiting for mgr epoch 9... 2024-01-13T08:57:30.827 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:30 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:57:30.379999+0000 mon.smithi077 (mon.0) 71 : cluster [INF] Active manager daemon smithi077.amvfyi restarted 2024-01-13T08:57:30.827 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:30 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:57:30.380220+0000 mon.smithi077 (mon.0) 72 : cluster [INF] Activating manager daemon smithi077.amvfyi 2024-01-13T08:57:30.827 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:30 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:57:30.383059+0000 mon.smithi077 (mon.0) 73 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-01-13T08:57:30.827 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:30 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:57:30.383174+0000 mon.smithi077 (mon.0) 74 : cluster [DBG] mgrmap e10: smithi077.amvfyi(active, starting, since 0.00304151s) 2024-01-13T08:57:30.827 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:30 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:30.383979+0000 mon.smithi077 (mon.0) 75 : audit [DBG] from='mgr.14156 172.21.15.77:0/48863403' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-01-13T08:57:30.828 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:30 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:30.384657+0000 mon.smithi077 (mon.0) 76 : audit [DBG] from='mgr.14156 172.21.15.77:0/48863403' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mgr metadata", "who": "smithi077.amvfyi", "id": "smithi077.amvfyi"}]: dispatch 2024-01-13T08:57:30.828 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:30 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:30.385288+0000 mon.smithi077 (mon.0) 77 : audit [DBG] from='mgr.14156 172.21.15.77:0/48863403' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-13T08:57:30.828 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:30 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:30.385496+0000 mon.smithi077 (mon.0) 78 : audit [DBG] from='mgr.14156 172.21.15.77:0/48863403' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-13T08:57:30.828 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:30 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:30.385680+0000 mon.smithi077 (mon.0) 79 : audit [DBG] from='mgr.14156 172.21.15.77:0/48863403' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-13T08:57:30.828 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:30 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:57:30.399583+0000 mon.smithi077 (mon.0) 80 : cluster [INF] Manager daemon smithi077.amvfyi is now available 2024-01-13T08:57:31.386 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: { 2024-01-13T08:57:31.386 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-01-13T08:57:31.386 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "initialized": true 2024-01-13T08:57:31.386 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: } 2024-01-13T08:57:31.720 INFO:teuthology.orchestra.run.smithi077.stderr:mgr epoch 9 is available 2024-01-13T08:57:31.720 INFO:teuthology.orchestra.run.smithi077.stderr:Generating a dashboard self-signed certificate... 2024-01-13T08:57:31.801 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:31 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:30.493220+0000 mon.smithi077 (mon.0) 81 : audit [DBG] from='mgr.14156 172.21.15.77:0/48863403' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:57:31.802 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:31 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:30.494675+0000 mon.smithi077 (mon.0) 82 : audit [DBG] from='mgr.14156 172.21.15.77:0/48863403' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:57:31.802 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:31 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:30.519362+0000 mon.smithi077 (mon.0) 83 : audit [INF] from='mgr.14156 172.21.15.77:0/48863403' entity='mgr.smithi077.amvfyi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.amvfyi/mirror_snapshot_schedule"}]: dispatch 2024-01-13T08:57:31.802 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:31 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:30.522255+0000 mon.smithi077 (mon.0) 84 : audit [INF] from='mgr.14156 172.21.15.77:0/48863403' entity='mgr.smithi077.amvfyi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.amvfyi/trash_purge_schedule"}]: dispatch 2024-01-13T08:57:31.802 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:31 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:31.315435+0000 mon.smithi077 (mon.0) 85 : audit [INF] from='mgr.14156 172.21.15.77:0/48863403' entity='mgr.smithi077.amvfyi' 2024-01-13T08:57:31.802 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:31 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:57:31.387508+0000 mon.smithi077 (mon.0) 86 : cluster [DBG] mgrmap e11: smithi077.amvfyi(active, since 1.00737s) 2024-01-13T08:57:32.496 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: Self-signed certificate created 2024-01-13T08:57:32.828 INFO:teuthology.orchestra.run.smithi077.stderr:Creating initial admin user... 2024-01-13T08:57:33.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:32 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:31.386814+0000 mgr.smithi077.amvfyi (mgr.14156) 1 : audit [DBG] from='client.14160 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-13T08:57:33.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:32 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:31.393099+0000 mgr.smithi077.amvfyi (mgr.14156) 2 : audit [DBG] from='client.14160 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-13T08:57:33.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:32 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:32.501498+0000 mon.smithi077 (mon.0) 87 : audit [INF] from='mgr.14156 172.21.15.77:0/48863403' entity='mgr.smithi077.amvfyi' 2024-01-13T08:57:33.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:32 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:32.503370+0000 mon.smithi077 (mon.0) 88 : audit [INF] from='mgr.14156 172.21.15.77:0/48863403' entity='mgr.smithi077.amvfyi' 2024-01-13T08:57:33.581 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$Q6hqvM1dMhftWa0ckPKe7uM7bC55JbX82Q0YlvQH8MKhdc8O86v52", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1705136253, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-01-13T08:57:33.920 INFO:teuthology.orchestra.run.smithi077.stderr:Fetching dashboard port number... 2024-01-13T08:57:34.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:32.430887+0000 mgr.smithi077.amvfyi (mgr.14156) 3 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T08:57:34.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:57:32.924949+0000 mon.smithi077 (mon.0) 89 : cluster [DBG] mgrmap e12: smithi077.amvfyi(active, since 2s) 2024-01-13T08:57:34.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:33.588366+0000 mon.smithi077 (mon.0) 90 : audit [INF] from='mgr.14156 172.21.15.77:0/48863403' entity='mgr.smithi077.amvfyi' 2024-01-13T08:57:34.414 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 8443 2024-01-13T08:57:34.778 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: disabled 2024-01-13T08:57:34.785 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: inactive 2024-01-13T08:57:34.786 INFO:teuthology.orchestra.run.smithi077.stderr:firewalld.service is not enabled 2024-01-13T08:57:34.786 INFO:teuthology.orchestra.run.smithi077.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-01-13T08:57:34.788 INFO:teuthology.orchestra.run.smithi077.stderr:Ceph Dashboard is now available at: 2024-01-13T08:57:34.788 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-01-13T08:57:34.788 INFO:teuthology.orchestra.run.smithi077.stderr: URL: https://smithi077:8443/ 2024-01-13T08:57:34.788 INFO:teuthology.orchestra.run.smithi077.stderr: User: admin 2024-01-13T08:57:34.788 INFO:teuthology.orchestra.run.smithi077.stderr: Password: av66bjz7o0 2024-01-13T08:57:34.788 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-01-13T08:57:34.788 INFO:teuthology.orchestra.run.smithi077.stderr:Enabling autotune for osd_memory_target 2024-01-13T08:57:35.313 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:33.371434+0000 mgr.smithi077.amvfyi (mgr.14156) 4 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T08:57:35.313 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:34.421305+0000 mon.smithi077 (mon.0) 91 : audit [DBG] from='client.? 172.21.15.77:0/2552193607' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-01-13T08:57:36.231 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-01-13T08:57:36.595 INFO:teuthology.orchestra.run.smithi077.stderr:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-01-13T08:57:36.595 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-01-13T08:57:36.595 INFO:teuthology.orchestra.run.smithi077.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-01-13T08:57:36.595 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-01-13T08:57:36.595 INFO:teuthology.orchestra.run.smithi077.stderr:Or, if you are only running a single cluster on this host: 2024-01-13T08:57:36.595 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-01-13T08:57:36.595 INFO:teuthology.orchestra.run.smithi077.stderr: sudo /home/ubuntu/cephtest/cephadm shell 2024-01-13T08:57:36.595 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-01-13T08:57:36.595 INFO:teuthology.orchestra.run.smithi077.stderr:Please consider enabling telemetry to help improve Ceph: 2024-01-13T08:57:36.595 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-01-13T08:57:36.596 INFO:teuthology.orchestra.run.smithi077.stderr: ceph telemetry on 2024-01-13T08:57:36.596 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-01-13T08:57:36.596 INFO:teuthology.orchestra.run.smithi077.stderr:For more information see: 2024-01-13T08:57:36.596 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-01-13T08:57:36.596 INFO:teuthology.orchestra.run.smithi077.stderr: https://docs.ceph.com/en/pacific/mgr/telemetry/ 2024-01-13T08:57:36.596 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-01-13T08:57:36.596 INFO:teuthology.orchestra.run.smithi077.stderr:Bootstrap complete. 2024-01-13T08:57:36.632 INFO:tasks.cephadm:Fetching config... 2024-01-13T08:57:36.632 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-13T08:57:36.632 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-01-13T08:57:36.649 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-01-13T08:57:36.649 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-13T08:57:36.650 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-01-13T08:57:36.707 INFO:tasks.cephadm:Fetching mon keyring... 2024-01-13T08:57:36.707 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-13T08:57:36.707 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi077/keyring of=/dev/stdout 2024-01-13T08:57:36.777 INFO:tasks.cephadm:Fetching pub ssh key... 2024-01-13T08:57:36.777 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-13T08:57:36.777 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-01-13T08:57:36.836 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-01-13T08:57:36.836 DEBUG:teuthology.orchestra.run.smithi077:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtBLew986nMMKO+PX4BFHdncEo3FM0poW9t5aedrqMiAAzujcsX+aYPOFABULx8GRD0HFxEKy5yB54lkLksgDigE4re+mDRXl8ecYoMk6ol6+jH5cDiNR9U5O8V2PJaNIzFn0awNv4U81SaBVZcbIhVnMhhRMgm8uYW45KizJnffZ3ftP9wOSgPT6fnp9UPzOR7gvqmPat3DIEmUVPFmwHMsni/Bkv8aVwWVC80yUjzxuCO6gh65bNSntJxfP449FVlsOcMCxkHbvgDYJHc9YQ3uhZ11HFPm2gP5mCEHTOUTeNZyyTVVUAG8Q/boEUm/xi8tdF2MbEO2i7EJR7tjyX1ZLkPBysIGqs5Ie1nOcuSFQs05dC6Vh9gc4TFnasApc1Zc9JR/Ik82YC5eRHvD4vdAjSzZgvGtUiAEOXP8ADbfgxgdOJvnFRrJLmiGwqA1A9gfk/Whkie2UpoyJXX8EnyOvhrGd3fAqfbED+BrqPGLwmsYhlQ4Lc7JGKaIJKOgM= ceph-99e41a38-b1f1-11ee-95ac-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-13T08:57:36.914 INFO:teuthology.orchestra.run.smithi077.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtBLew986nMMKO+PX4BFHdncEo3FM0poW9t5aedrqMiAAzujcsX+aYPOFABULx8GRD0HFxEKy5yB54lkLksgDigE4re+mDRXl8ecYoMk6ol6+jH5cDiNR9U5O8V2PJaNIzFn0awNv4U81SaBVZcbIhVnMhhRMgm8uYW45KizJnffZ3ftP9wOSgPT6fnp9UPzOR7gvqmPat3DIEmUVPFmwHMsni/Bkv8aVwWVC80yUjzxuCO6gh65bNSntJxfP449FVlsOcMCxkHbvgDYJHc9YQ3uhZ11HFPm2gP5mCEHTOUTeNZyyTVVUAG8Q/boEUm/xi8tdF2MbEO2i7EJR7tjyX1ZLkPBysIGqs5Ie1nOcuSFQs05dC6Vh9gc4TFnasApc1Zc9JR/Ik82YC5eRHvD4vdAjSzZgvGtUiAEOXP8ADbfgxgdOJvnFRrJLmiGwqA1A9gfk/Whkie2UpoyJXX8EnyOvhrGd3fAqfbED+BrqPGLwmsYhlQ4Lc7JGKaIJKOgM= ceph-99e41a38-b1f1-11ee-95ac-87774f69a715 2024-01-13T08:57:36.927 DEBUG:teuthology.orchestra.run.smithi141:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtBLew986nMMKO+PX4BFHdncEo3FM0poW9t5aedrqMiAAzujcsX+aYPOFABULx8GRD0HFxEKy5yB54lkLksgDigE4re+mDRXl8ecYoMk6ol6+jH5cDiNR9U5O8V2PJaNIzFn0awNv4U81SaBVZcbIhVnMhhRMgm8uYW45KizJnffZ3ftP9wOSgPT6fnp9UPzOR7gvqmPat3DIEmUVPFmwHMsni/Bkv8aVwWVC80yUjzxuCO6gh65bNSntJxfP449FVlsOcMCxkHbvgDYJHc9YQ3uhZ11HFPm2gP5mCEHTOUTeNZyyTVVUAG8Q/boEUm/xi8tdF2MbEO2i7EJR7tjyX1ZLkPBysIGqs5Ie1nOcuSFQs05dC6Vh9gc4TFnasApc1Zc9JR/Ik82YC5eRHvD4vdAjSzZgvGtUiAEOXP8ADbfgxgdOJvnFRrJLmiGwqA1A9gfk/Whkie2UpoyJXX8EnyOvhrGd3fAqfbED+BrqPGLwmsYhlQ4Lc7JGKaIJKOgM= ceph-99e41a38-b1f1-11ee-95ac-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-13T08:57:36.975 INFO:teuthology.orchestra.run.smithi141.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtBLew986nMMKO+PX4BFHdncEo3FM0poW9t5aedrqMiAAzujcsX+aYPOFABULx8GRD0HFxEKy5yB54lkLksgDigE4re+mDRXl8ecYoMk6ol6+jH5cDiNR9U5O8V2PJaNIzFn0awNv4U81SaBVZcbIhVnMhhRMgm8uYW45KizJnffZ3ftP9wOSgPT6fnp9UPzOR7gvqmPat3DIEmUVPFmwHMsni/Bkv8aVwWVC80yUjzxuCO6gh65bNSntJxfP449FVlsOcMCxkHbvgDYJHc9YQ3uhZ11HFPm2gP5mCEHTOUTeNZyyTVVUAG8Q/boEUm/xi8tdF2MbEO2i7EJR7tjyX1ZLkPBysIGqs5Ie1nOcuSFQs05dC6Vh9gc4TFnasApc1Zc9JR/Ik82YC5eRHvD4vdAjSzZgvGtUiAEOXP8ADbfgxgdOJvnFRrJLmiGwqA1A9gfk/Whkie2UpoyJXX8EnyOvhrGd3fAqfbED+BrqPGLwmsYhlQ4Lc7JGKaIJKOgM= ceph-99e41a38-b1f1-11ee-95ac-87774f69a715 2024-01-13T08:57:36.988 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-01-13T08:57:37.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:36.236470+0000 mon.smithi077 (mon.0) 92 : audit [INF] from='client.? 172.21.15.77:0/1227149495' entity='client.admin' 2024-01-13T08:57:38.443 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:38 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:57:37.238375+0000 mon.smithi077 (mon.0) 93 : cluster [DBG] mgrmap e13: smithi077.amvfyi(active, since 6s) 2024-01-13T08:57:39.880 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-01-13T08:57:39.880 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-01-13T08:57:40.246 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:39 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:38.946495+0000 mon.smithi077 (mon.0) 94 : audit [INF] from='mgr.14156 172.21.15.77:0/48863403' entity='mgr.smithi077.amvfyi' 2024-01-13T08:57:40.246 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:39 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:39.337616+0000 mon.smithi077 (mon.0) 95 : audit [INF] from='mgr.14156 172.21.15.77:0/48863403' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config rm", "who": "osd/host:smithi077", "name": "osd_memory_target"}]: dispatch 2024-01-13T08:57:40.246 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:39 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:39.341188+0000 mon.smithi077 (mon.0) 96 : audit [INF] from='mgr.14156 172.21.15.77:0/48863403' entity='mgr.smithi077.amvfyi' 2024-01-13T08:57:40.246 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:39 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:39.553122+0000 mon.smithi077 (mon.0) 97 : audit [INF] from='client.? 172.21.15.77:0/90123043' entity='client.admin' 2024-01-13T08:57:40.976 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:57:39.351828+0000 mgr.smithi077.amvfyi (mgr.14156) 5 : cephadm [INF] Deploying daemon alertmanager.smithi077 on smithi077 2024-01-13T08:57:41.717 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi141 2024-01-13T08:57:41.717 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-13T08:57:41.717 DEBUG:teuthology.orchestra.run.smithi141:> dd of=/etc/ceph/ceph.conf 2024-01-13T08:57:41.738 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-13T08:57:41.738 DEBUG:teuthology.orchestra.run.smithi141:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-13T08:57:41.798 INFO:tasks.cephadm:Adding host smithi141 to orchestrator... 2024-01-13T08:57:41.799 DEBUG:teuthology.orchestra.run.smithi141:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph orch host add smithi141 2024-01-13T08:57:42.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:42 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:41.344389+0000 mgr.smithi077.amvfyi (mgr.14156) 6 : audit [DBG] from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T08:57:42.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:42 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:41.346820+0000 mon.smithi077 (mon.0) 98 : audit [INF] from='mgr.14156 172.21.15.77:0/48863403' entity='mgr.smithi077.amvfyi' 2024-01-13T08:57:48.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:48 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:47.208212+0000 mon.smithi077 (mon.0) 99 : audit [INF] from='mgr.14156 172.21.15.77:0/48863403' entity='mgr.smithi077.amvfyi' 2024-01-13T08:57:48.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:48 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:47.209776+0000 mon.smithi077 (mon.0) 100 : audit [INF] from='mgr.14156 172.21.15.77:0/48863403' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi077", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-13T08:57:48.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:48 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:47.211314+0000 mon.smithi077 (mon.0) 101 : audit [INF] from='mgr.14156 172.21.15.77:0/48863403' entity='mgr.smithi077.amvfyi' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi077", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-01-13T08:57:48.577 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:48 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:47.211815+0000 mon.smithi077 (mon.0) 102 : audit [DBG] from='mgr.14156 172.21.15.77:0/48863403' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:57:48.577 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:48 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:57:47.212266+0000 mgr.smithi077.amvfyi (mgr.14156) 7 : cephadm [INF] Deploying daemon crash.smithi077 on smithi077 2024-01-13T08:57:50.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:49 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:48.741500+0000 mon.smithi077 (mon.0) 103 : audit [INF] from='mgr.14156 172.21.15.77:0/48863403' entity='mgr.smithi077.amvfyi' 2024-01-13T08:57:50.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:49 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:48.870972+0000 mon.smithi077 (mon.0) 104 : audit [INF] from='mgr.14156 172.21.15.77:0/48863403' entity='mgr.smithi077.amvfyi' 2024-01-13T08:57:50.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:49 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:48.873440+0000 mon.smithi077 (mon.0) 105 : audit [INF] from='mgr.14156 172.21.15.77:0/48863403' entity='mgr.smithi077.amvfyi' 2024-01-13T08:57:50.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:49 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:48.875352+0000 mon.smithi077 (mon.0) 106 : audit [INF] from='mgr.14156 172.21.15.77:0/48863403' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-13T08:57:50.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:49 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:48.875721+0000 mgr.smithi077.amvfyi (mgr.14156) 8 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-13T08:57:50.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:49 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:48.878860+0000 mon.smithi077 (mon.0) 107 : audit [INF] from='mgr.14156 172.21.15.77:0/48863403' entity='mgr.smithi077.amvfyi' 2024-01-13T08:57:50.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:49 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:57:48.886378+0000 mgr.smithi077.amvfyi (mgr.14156) 9 : cephadm [INF] Deploying daemon grafana.smithi077 on smithi077 2024-01-13T08:57:51.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:51 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:57:50.386448+0000 mgr.smithi077.amvfyi (mgr.14156) 10 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:57:51.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:51 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:50.518597+0000 mon.smithi077 (mon.0) 108 : audit [INF] from='mgr.14156 172.21.15.77:0/48863403' entity='mgr.smithi077.amvfyi' 2024-01-13T08:57:54.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:57:52.386671+0000 mgr.smithi077.amvfyi (mgr.14156) 11 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:57:56.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:55 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:57:54.386888+0000 mgr.smithi077.amvfyi (mgr.14156) 12 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:57:58.299 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:57 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:57:56.387121+0000 mgr.smithi077.amvfyi (mgr.14156) 13 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:58:00.086 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:59 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:57:58.387416+0000 mgr.smithi077.amvfyi (mgr.14156) 14 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:58:00.086 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:57:59 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:57:59.782327+0000 mon.smithi077 (mon.0) 109 : audit [INF] from='mgr.14156 172.21.15.77:0/48863403' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:01.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:57:59.784329+0000 mgr.smithi077.amvfyi (mgr.14156) 15 : cephadm [INF] Deploying daemon node-exporter.smithi077 on smithi077 2024-01-13T08:58:01.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:00.522448+0000 mon.smithi077 (mon.0) 110 : audit [INF] from='mgr.14156 172.21.15.77:0/48863403' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:02.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:01 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:00.387922+0000 mgr.smithi077.amvfyi (mgr.14156) 16 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:58:03.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:03 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:02.111514+0000 mon.smithi077 (mon.0) 111 : audit [INF] from='mgr.14156 172.21.15.77:0/48863403' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:03.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:03 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:02.113546+0000 mon.smithi077 (mon.0) 112 : audit [INF] from='mgr.14156 172.21.15.77:0/48863403' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-01-13T08:58:03.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:03 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:02.388184+0000 mgr.smithi077.amvfyi (mgr.14156) 17 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:58:04.447 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:04 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:03.117014+0000 mon.smithi077 (mon.0) 113 : audit [INF] from='mgr.14156 172.21.15.77:0/48863403' entity='mgr.smithi077.amvfyi' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-01-13T08:58:04.447 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:04 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:03.117079+0000 mon.smithi077 (mon.0) 114 : cluster [DBG] mgrmap e14: smithi077.amvfyi(active, since 32s) 2024-01-13T08:58:08.676 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:08 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:08.388968+0000 mon.smithi077 (mon.0) 115 : cluster [INF] Active manager daemon smithi077.amvfyi restarted 2024-01-13T08:58:08.676 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:08 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:08.389188+0000 mon.smithi077 (mon.0) 116 : cluster [INF] Activating manager daemon smithi077.amvfyi 2024-01-13T08:58:08.677 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:08 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:08.391904+0000 mon.smithi077 (mon.0) 117 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-01-13T08:58:08.677 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:08 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:08.391993+0000 mon.smithi077 (mon.0) 118 : cluster [DBG] mgrmap e15: smithi077.amvfyi(active, starting, since 0.00289254s) 2024-01-13T08:58:08.677 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:08 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:08.393102+0000 mon.smithi077 (mon.0) 119 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-01-13T08:58:08.677 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:08 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:08.393668+0000 mon.smithi077 (mon.0) 120 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mgr metadata", "who": "smithi077.amvfyi", "id": "smithi077.amvfyi"}]: dispatch 2024-01-13T08:58:08.677 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:08 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:08.394288+0000 mon.smithi077 (mon.0) 121 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-13T08:58:08.677 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:08 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:08.394437+0000 mon.smithi077 (mon.0) 122 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-13T08:58:08.677 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:08 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:08.394563+0000 mon.smithi077 (mon.0) 123 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-13T08:58:08.677 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:08 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:08.408750+0000 mon.smithi077 (mon.0) 124 : cluster [INF] Manager daemon smithi077.amvfyi is now available 2024-01-13T08:58:09.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:09 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:08.504272+0000 mon.smithi077 (mon.0) 125 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:58:09.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:09 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:08.505722+0000 mon.smithi077 (mon.0) 126 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:58:09.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:09 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:08.529689+0000 mon.smithi077 (mon.0) 127 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:09.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:09 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:08.541600+0000 mon.smithi077 (mon.0) 128 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.amvfyi/mirror_snapshot_schedule"}]: dispatch 2024-01-13T08:58:09.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:09 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:08.545797+0000 mon.smithi077 (mon.0) 129 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.amvfyi/trash_purge_schedule"}]: dispatch 2024-01-13T08:58:09.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:09 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:09.393784+0000 mon.smithi077 (mon.0) 130 : cluster [DBG] mgrmap e16: smithi077.amvfyi(active, since 1.00468s) 2024-01-13T08:58:11.679 INFO:teuthology.orchestra.run.smithi141.stdout:Added host 'smithi141' with addr '172.21.15.141' 2024-01-13T08:58:11.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:11 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:10.445433+0000 mon.smithi077 (mon.0) 131 : cluster [DBG] mgrmap e17: smithi077.amvfyi(active, since 2s) 2024-01-13T08:58:11.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:11 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:11.074121+0000 mgr.smithi077.amvfyi (mgr.14180) 1 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi141", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T08:58:12.008 DEBUG:teuthology.orchestra.run.smithi141:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph orch host ls --format=json 2024-01-13T08:58:13.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:12 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:11.684752+0000 mon.smithi077 (mon.0) 132 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:13.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:12 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:58:11.685038+0000 mgr.smithi077.amvfyi (mgr.14180) 2 : cephadm [INF] Added host smithi141 2024-01-13T08:58:13.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:12 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:11.829840+0000 mon.smithi077 (mon.0) 133 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:13.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:12 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:12.111855+0000 mon.smithi077 (mon.0) 134 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config rm", "who": "osd/host:smithi077", "name": "osd_memory_target"}]: dispatch 2024-01-13T08:58:13.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:12 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:12.112680+0000 mon.smithi077 (mon.0) 135 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:58:13.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:12 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:12.113254+0000 mon.smithi077 (mon.0) 136 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T08:58:13.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:12 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:58:12.114287+0000 mgr.smithi077.amvfyi (mgr.14180) 3 : cephadm [INF] Updating smithi077:/etc/ceph/ceph.conf 2024-01-13T08:58:13.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:12 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:58:12.134936+0000 mgr.smithi077.amvfyi (mgr.14180) 4 : cephadm [INF] Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-01-13T08:58:13.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:12 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:12.163318+0000 mon.smithi077 (mon.0) 137 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:13.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:12 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:12.166315+0000 mon.smithi077 (mon.0) 138 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:13.233 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-13T08:58:13.233 INFO:teuthology.orchestra.run.smithi141.stdout:[{"addr": "172.21.15.77", "hostname": "smithi077", "labels": [], "status": ""}, {"addr": "172.21.15.141", "hostname": "smithi141", "labels": [], "status": ""}] 2024-01-13T08:58:13.601 INFO:tasks.cephadm:Setting crush tunables to default 2024-01-13T08:58:13.601 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd crush tunables default 2024-01-13T08:58:14.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:13 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:58:12.177366+0000 mgr.smithi077.amvfyi (mgr.14180) 5 : cephadm [INF] Deploying daemon prometheus.smithi077 on smithi077 2024-01-13T08:58:15.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:13.239553+0000 mgr.smithi077.amvfyi (mgr.14180) 6 : audit [DBG] from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-13T08:58:15.925 INFO:teuthology.orchestra.run.smithi077.stderr:adjusted tunables profile to default 2024-01-13T08:58:16.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:15.877168+0000 mon.smithi077 (mon.0) 139 : audit [INF] from='client.? 172.21.15.77:0/1708011291' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-01-13T08:58:17.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:16 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:15.929585+0000 mon.smithi077 (mon.0) 140 : audit [INF] from='client.? 172.21.15.77:0/1708011291' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-01-13T08:58:17.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:16 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:15.929652+0000 mon.smithi077 (mon.0) 141 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-01-13T08:58:17.487 INFO:tasks.cephadm:Adding mon.smithi077 on smithi077 2024-01-13T08:58:17.487 INFO:tasks.cephadm:Adding mon.smithi141 on smithi141 2024-01-13T08:58:17.487 DEBUG:teuthology.orchestra.run.smithi141:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph orch apply mon '2;smithi077:172.21.15.77=smithi077;smithi141:172.21.15.141=smithi141' 2024-01-13T08:58:18.766 INFO:teuthology.orchestra.run.smithi141.stdout:Scheduled mon update... 2024-01-13T08:58:19.148 DEBUG:teuthology.orchestra.run.smithi141:mon.smithi141> sudo journalctl -f -n 0 -u ceph-99e41a38-b1f1-11ee-95ac-87774f69a715@mon.smithi141.service 2024-01-13T08:58:19.150 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T08:58:19.150 DEBUG:teuthology.orchestra.run.smithi141:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T08:58:19.178 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:-- Logs begin at Sat 2024-01-13 08:46:23 UTC. -- 2024-01-13T08:58:20.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:19 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:18.768673+0000 mgr.smithi077.amvfyi (mgr.14180) 7 : audit [DBG] from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi077:172.21.15.77=smithi077;smithi141:172.21.15.141=smithi141", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T08:58:20.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:19 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:58:18.770578+0000 mgr.smithi077.amvfyi (mgr.14180) 8 : cephadm [INF] Saving service mon spec with placement smithi077:172.21.15.77=smithi077;smithi141:172.21.15.141=smithi141;count:2 2024-01-13T08:58:20.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:19 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:18.773080+0000 mon.smithi077 (mon.0) 142 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:20.487 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-13T08:58:20.487 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":1,"fsid":"99e41a38-b1f1-11ee-95ac-87774f69a715","modified":"2024-01-13T08:56:41.569521Z","created":"2024-01-13T08:56:41.569521Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T08:58:20.489 INFO:teuthology.orchestra.run.smithi141.stderr:dumped monmap epoch 1 2024-01-13T08:58:21.820 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T08:58:21.820 DEBUG:teuthology.orchestra.run.smithi141:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T08:58:21.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:20.354987+0000 mon.smithi077 (mon.0) 143 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:21.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:20.356020+0000 mon.smithi077 (mon.0) 144 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:58:21.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:20.494485+0000 mon.smithi077 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.141:0/1177226149' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T08:58:21.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:20.814020+0000 mon.smithi077 (mon.0) 146 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:21.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:21.089180+0000 mon.smithi077 (mon.0) 147 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:23.523 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-13T08:58:23.523 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":1,"fsid":"99e41a38-b1f1-11ee-95ac-87774f69a715","modified":"2024-01-13T08:56:41.569521Z","created":"2024-01-13T08:56:41.569521Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T08:58:23.525 INFO:teuthology.orchestra.run.smithi141.stderr:dumped monmap epoch 1 2024-01-13T08:58:24.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:23 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:22.739804+0000 mon.smithi077 (mon.0) 148 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:24.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:23 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:23.530831+0000 mon.smithi077 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.141:0/2243594400' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T08:58:24.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:23 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:23.535658+0000 mon.smithi077 (mon.0) 150 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:24.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:23 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:23.606778+0000 mon.smithi077 (mon.0) 151 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:24.901 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T08:58:24.901 DEBUG:teuthology.orchestra.run.smithi141:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T08:58:26.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:25 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:24.845295+0000 mon.smithi077 (mon.0) 152 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:26.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:25 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:24.846183+0000 mon.smithi077 (mon.0) 153 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config rm", "who": "osd/host:smithi141", "name": "osd_memory_target"}]: dispatch 2024-01-13T08:58:26.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:25 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:24.847240+0000 mon.smithi077 (mon.0) 154 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:58:26.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:25 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:24.847992+0000 mon.smithi077 (mon.0) 155 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T08:58:26.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:25 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:58:24.849178+0000 mgr.smithi077.amvfyi (mgr.14180) 9 : cephadm [INF] Updating smithi141:/etc/ceph/ceph.conf 2024-01-13T08:58:26.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:25 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:25.567352+0000 mon.smithi077 (mon.0) 156 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:26.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:25 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:25.570155+0000 mon.smithi077 (mon.0) 157 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:26.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:25 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:25.571787+0000 mon.smithi077 (mon.0) 158 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi141", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-13T08:58:26.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:25 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:25.573836+0000 mon.smithi077 (mon.0) 159 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi141", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-01-13T08:58:26.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:25 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:25.574639+0000 mon.smithi077 (mon.0) 160 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:58:27.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:26 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:58:25.246068+0000 mgr.smithi077.amvfyi (mgr.14180) 10 : cephadm [INF] Updating smithi141:/etc/ceph/ceph.client.admin.keyring 2024-01-13T08:58:27.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:26 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:58:25.575336+0000 mgr.smithi077.amvfyi (mgr.14180) 11 : cephadm [INF] Deploying daemon crash.smithi141 on smithi141 2024-01-13T08:58:28.158 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-13T08:58:28.158 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":1,"fsid":"99e41a38-b1f1-11ee-95ac-87774f69a715","modified":"2024-01-13T08:56:41.569521Z","created":"2024-01-13T08:56:41.569521Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T08:58:28.160 INFO:teuthology.orchestra.run.smithi141.stderr:dumped monmap epoch 1 2024-01-13T08:58:28.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:28.165524+0000 mon.smithi077 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.141:0/3447593309' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T08:58:29.540 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T08:58:29.541 DEBUG:teuthology.orchestra.run.smithi141:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T08:58:30.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:28.395601+0000 mgr.smithi077.amvfyi (mgr.14180) 12 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:58:30.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:28.802905+0000 mon.smithi077 (mon.0) 162 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:30.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:28.805093+0000 mon.smithi077 (mon.0) 163 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi141.ujrxla", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-13T08:58:30.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:28.807162+0000 mon.smithi077 (mon.0) 164 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi141.ujrxla", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-01-13T08:58:30.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:28.807901+0000 mon.smithi077 (mon.0) 165 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-13T08:58:30.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:28.808911+0000 mon.smithi077 (mon.0) 166 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:58:30.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:58:28.809591+0000 mgr.smithi077.amvfyi (mgr.14180) 13 : cephadm [INF] Deploying daemon mgr.smithi141.ujrxla on smithi141 2024-01-13T08:58:31.171 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-13T08:58:31.171 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":1,"fsid":"99e41a38-b1f1-11ee-95ac-87774f69a715","modified":"2024-01-13T08:56:41.569521Z","created":"2024-01-13T08:56:41.569521Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T08:58:31.175 INFO:teuthology.orchestra.run.smithi141.stderr:dumped monmap epoch 1 2024-01-13T08:58:31.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:31 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:30.395957+0000 mgr.smithi077.amvfyi (mgr.14180) 14 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:58:31.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:31 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:30.447419+0000 mon.smithi077 (mon.0) 167 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:31.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:31 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:30.449448+0000 mon.smithi077 (mon.0) 168 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-13T08:58:31.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:31 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:30.450194+0000 mon.smithi077 (mon.0) 169 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:58:31.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:31 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:58:30.450841+0000 mgr.smithi077.amvfyi (mgr.14180) 15 : cephadm [INF] Deploying daemon mon.smithi141 on smithi141 2024-01-13T08:58:31.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:31 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:31.178587+0000 mon.smithi077 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.141:0/2233958120' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T08:58:32.638 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T08:58:32.638 DEBUG:teuthology.orchestra.run.smithi141:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T08:58:32.813 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 systemd[1]: Starting Ceph mon.smithi141 for 99e41a38-b1f1-11ee-95ac-87774f69a715... 2024-01-13T08:58:33.130 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.879+0000 7f6343dbc880 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-13T08:58:33.130 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.879+0000 7f6343dbc880 0 ceph version 16.2.14-456-g445562ab (445562ab4bc3ddfb386936119050695810860bcb) pacific (stable), process ceph-mon, pid 7 2024-01-13T08:58:33.130 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.879+0000 7f6343dbc880 0 pidfile_write: ignore empty --pid-file 2024-01-13T08:58:33.130 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 bash[34398]: 739db66f55e3f12d7f123f8cb507d5f7b0912462372b77ba180021606fc9de54 2024-01-13T08:58:33.130 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.884+0000 7f6343dbc880 0 load: jerasure load: lrc load: isa 2024-01-13T08:58:33.131 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: RocksDB version: 6.8.1 2024-01-13T08:58:33.131 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: 2024-01-13T08:58:33.131 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-13T08:58:33.131 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Compile date Jan 12 2024 2024-01-13T08:58:33.131 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: DB SUMMARY 2024-01-13T08:58:33.131 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: 2024-01-13T08:58:33.131 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: CURRENT file: CURRENT 2024-01-13T08:58:33.131 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: 2024-01-13T08:58:33.131 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: IDENTITY file: IDENTITY 2024-01-13T08:58:33.131 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: 2024-01-13T08:58:33.131 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: MANIFEST file: MANIFEST-000001 size: 13 Bytes 2024-01-13T08:58:33.131 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: 2024-01-13T08:58:33.131 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi141/store.db dir, Total Num: 0, files: 2024-01-13T08:58:33.132 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: 2024-01-13T08:58:33.132 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi141/store.db: 000003.log size: 511 ; 2024-01-13T08:58:33.132 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: 2024-01-13T08:58:33.132 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.error_if_exists: 0 2024-01-13T08:58:33.132 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.create_if_missing: 0 2024-01-13T08:58:33.132 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.paranoid_checks: 1 2024-01-13T08:58:33.132 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.env: 0x55bc0ebdc080 2024-01-13T08:58:33.132 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.fs: Posix File System 2024-01-13T08:58:33.132 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.info_log: 0x55bc10aa7d20 2024-01-13T08:58:33.132 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-13T08:58:33.132 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.statistics: (nil) 2024-01-13T08:58:33.132 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.use_fsync: 0 2024-01-13T08:58:33.133 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.max_log_file_size: 0 2024-01-13T08:58:33.133 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-13T08:58:33.133 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-13T08:58:33.133 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-13T08:58:33.133 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-13T08:58:33.133 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.allow_fallocate: 1 2024-01-13T08:58:33.133 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-13T08:58:33.133 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-13T08:58:33.133 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.use_direct_reads: 0 2024-01-13T08:58:33.133 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-13T08:58:33.133 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.create_missing_column_families: 0 2024-01-13T08:58:33.133 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.db_log_dir: 2024-01-13T08:58:33.133 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi141/store.db 2024-01-13T08:58:33.134 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-13T08:58:33.134 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.max_subcompactions: 1 2024-01-13T08:58:33.134 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.max_background_flushes: -1 2024-01-13T08:58:33.134 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-13T08:58:33.134 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-13T08:58:33.134 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-13T08:58:33.134 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-13T08:58:33.134 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-13T08:58:33.134 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.advise_random_on_open: 1 2024-01-13T08:58:33.134 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-13T08:58:33.134 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.write_buffer_manager: 0x55bc116fe480 2024-01-13T08:58:33.134 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-13T08:58:33.135 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-13T08:58:33.135 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-13T08:58:33.135 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-13T08:58:33.135 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.rate_limiter: (nil) 2024-01-13T08:58:33.135 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-13T08:58:33.135 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-13T08:58:33.135 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-13T08:58:33.135 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-13T08:58:33.135 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.unordered_write: 0 2024-01-13T08:58:33.135 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-13T08:58:33.135 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-13T08:58:33.135 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-13T08:58:33.135 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-13T08:58:33.136 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.row_cache: None 2024-01-13T08:58:33.136 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.wal_filter: None 2024-01-13T08:58:33.136 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-13T08:58:33.136 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-13T08:58:33.136 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.preserve_deletes: 0 2024-01-13T08:58:33.136 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.two_write_queues: 0 2024-01-13T08:58:33.136 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.manual_wal_flush: 0 2024-01-13T08:58:33.136 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.atomic_flush: 0 2024-01-13T08:58:33.136 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-13T08:58:33.136 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-13T08:58:33.136 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-13T08:58:33.136 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.log_readahead_size: 0 2024-01-13T08:58:33.137 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-13T08:58:33.137 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.max_background_jobs: 2 2024-01-13T08:58:33.137 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.max_background_compactions: -1 2024-01-13T08:58:33.137 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-13T08:58:33.137 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-13T08:58:33.140 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-13T08:58:33.141 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.max_total_wal_size: 0 2024-01-13T08:58:33.141 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-13T08:58:33.141 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-13T08:58:33.141 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-13T08:58:33.141 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-13T08:58:33.141 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.max_open_files: -1 2024-01-13T08:58:33.141 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.bytes_per_sync: 0 2024-01-13T08:58:33.141 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-13T08:58:33.141 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-13T08:58:33.141 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-13T08:58:33.141 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Compression algorithms supported: 2024-01-13T08:58:33.141 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-13T08:58:33.141 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: kZSTD supported: 0 2024-01-13T08:58:33.141 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: kXpressCompression supported: 0 2024-01-13T08:58:33.142 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-13T08:58:33.142 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: kLZ4Compression supported: 1 2024-01-13T08:58:33.142 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: kBZip2Compression supported: 0 2024-01-13T08:58:33.142 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: kZlibCompression supported: 1 2024-01-13T08:58:33.142 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: kSnappyCompression supported: 1 2024-01-13T08:58:33.142 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-13T08:58:33.142 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi141/store.db/MANIFEST-000001 2024-01-13T08:58:33.142 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: 2024-01-13T08:58:33.142 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-13T08:58:33.142 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: 2024-01-13T08:58:33.142 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-13T08:58:33.142 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.merge_operator: 2024-01-13T08:58:33.142 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.compaction_filter: None 2024-01-13T08:58:33.142 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.compaction_filter_factory: None 2024-01-13T08:58:33.143 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-13T08:58:33.143 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-13T08:58:33.143 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55bc10a57d28) 2024-01-13T08:58:33.143 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cache_index_and_filter_blocks: 1 2024-01-13T08:58:33.143 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-13T08:58:33.143 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-13T08:58:33.143 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: pin_top_level_index_and_filter: 1 2024-01-13T08:58:33.143 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: index_type: 0 2024-01-13T08:58:33.143 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: data_block_index_type: 0 2024-01-13T08:58:33.143 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: index_shortening: 1 2024-01-13T08:58:33.143 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: data_block_hash_table_util_ratio: 0.750000 2024-01-13T08:58:33.143 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: hash_index_allow_collision: 1 2024-01-13T08:58:33.143 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: checksum: 1 2024-01-13T08:58:33.143 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: no_block_cache: 0 2024-01-13T08:58:33.144 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: block_cache: 0x55bc10a8cf10 2024-01-13T08:58:33.144 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: block_cache_name: BinnedLRUCache 2024-01-13T08:58:33.144 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: block_cache_options: 2024-01-13T08:58:33.144 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: capacity : 536870912 2024-01-13T08:58:33.144 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: num_shard_bits : 4 2024-01-13T08:58:33.144 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: strict_capacity_limit : 0 2024-01-13T08:58:33.144 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: high_pri_pool_ratio: 0.000 2024-01-13T08:58:33.144 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: block_cache_compressed: (nil) 2024-01-13T08:58:33.144 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: persistent_cache: (nil) 2024-01-13T08:58:33.144 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: block_size: 4096 2024-01-13T08:58:33.144 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: block_size_deviation: 10 2024-01-13T08:58:33.144 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: block_restart_interval: 16 2024-01-13T08:58:33.144 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: index_block_restart_interval: 1 2024-01-13T08:58:33.144 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: metadata_block_size: 4096 2024-01-13T08:58:33.144 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: partition_filters: 0 2024-01-13T08:58:33.145 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: use_delta_encoding: 1 2024-01-13T08:58:33.145 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-13T08:58:33.145 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: whole_key_filtering: 1 2024-01-13T08:58:33.145 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: verify_compression: 0 2024-01-13T08:58:33.145 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: read_amp_bytes_per_bit: 0 2024-01-13T08:58:33.145 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: format_version: 2 2024-01-13T08:58:33.145 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: enable_index_compression: 1 2024-01-13T08:58:33.145 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: block_align: 0 2024-01-13T08:58:33.145 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: 2024-01-13T08:58:33.145 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-13T08:58:33.145 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-13T08:58:33.145 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.compression: NoCompression 2024-01-13T08:58:33.145 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-13T08:58:33.146 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-13T08:58:33.146 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-13T08:58:33.146 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.num_levels: 7 2024-01-13T08:58:33.146 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-13T08:58:33.146 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-13T08:58:33.146 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-13T08:58:33.146 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-13T08:58:33.146 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-13T08:58:33.146 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-13T08:58:33.146 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-13T08:58:33.146 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-13T08:58:33.146 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-13T08:58:33.146 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-13T08:58:33.146 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.compression_opts.level: 32767 2024-01-13T08:58:33.146 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-13T08:58:33.147 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-13T08:58:33.147 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-13T08:58:33.147 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.compression_opts.enabled: false 2024-01-13T08:58:33.147 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-13T08:58:33.147 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.885+0000 7f6343dbc880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-13T08:58:33.147 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-13T08:58:33.147 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-13T08:58:33.147 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-13T08:58:33.147 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-13T08:58:33.147 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-13T08:58:33.147 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-13T08:58:33.147 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-13T08:58:33.147 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-13T08:58:33.147 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-13T08:58:33.147 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-13T08:58:33.148 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-13T08:58:33.148 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-13T08:58:33.148 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-13T08:58:33.148 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-13T08:58:33.148 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-13T08:58:33.148 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.arena_block_size: 4194304 2024-01-13T08:58:33.148 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-13T08:58:33.148 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-13T08:58:33.148 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-13T08:58:33.148 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-13T08:58:33.148 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-13T08:58:33.148 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-13T08:58:33.148 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-13T08:58:33.148 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-13T08:58:33.149 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-13T08:58:33.149 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-13T08:58:33.149 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-13T08:58:33.149 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-13T08:58:33.149 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-13T08:58:33.149 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-13T08:58:33.149 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.table_properties_collectors: 2024-01-13T08:58:33.149 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.inplace_update_support: 0 2024-01-13T08:58:33.149 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-13T08:58:33.149 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-13T08:58:33.149 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-13T08:58:33.149 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-13T08:58:33.149 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.bloom_locality: 0 2024-01-13T08:58:33.149 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.max_successive_merges: 0 2024-01-13T08:58:33.149 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-13T08:58:33.150 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-13T08:58:33.150 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.force_consistency_checks: 0 2024-01-13T08:58:33.150 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-13T08:58:33.150 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.ttl: 2592000 2024-01-13T08:58:33.150 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-13T08:58:33.150 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi141/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-13T08:58:33.150 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: 2024-01-13T08:58:33.150 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-01-13T08:58:33.150 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: 2024-01-13T08:58:33.150 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705136312887154, "job": 1, "event": "recovery_started", "log_files": [3]} 2024-01-13T08:58:33.150 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.886+0000 7f6343dbc880 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #3 mode 2 2024-01-13T08:58:33.150 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.887+0000 7f6343dbc880 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-01-13T08:58:33.150 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 systemd[1]: Started Ceph mon.smithi141 for 99e41a38-b1f1-11ee-95ac-87774f69a715. 2024-01-13T08:58:33.151 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.887+0000 7f6343dbc880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705136312887869, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 4, "file_size": 1437, "table_properties": {"data_size": 523, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1705136312, "oldest_key_time": 3, "file_creation_time": 0}} 2024-01-13T08:58:33.151 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.887+0000 7f6343dbc880 4 rocksdb: [version_set.cc:3826] Creating manifest 5 2024-01-13T08:58:33.151 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: 2024-01-13T08:58:33.151 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.888+0000 7f6343dbc880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705136312888594, "job": 1, "event": "recovery_finished"} 2024-01-13T08:58:33.151 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.889+0000 7f6343dbc880 4 rocksdb: DB pointer 0x55bc10b55800 2024-01-13T08:58:33.151 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.889+0000 7f632abb0700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-13T08:58:33.151 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.889+0000 7f632abb0700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-13T08:58:33.151 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: ** DB Stats ** 2024-01-13T08:58:33.151 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T08:58:33.151 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-13T08:58:33.151 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-13T08:58:33.151 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-13T08:58:33.151 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-13T08:58:33.151 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-13T08:58:33.151 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-13T08:58:33.152 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: 2024-01-13T08:58:33.152 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: ** Compaction Stats [default] ** 2024-01-13T08:58:33.152 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T08:58:33.152 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T08:58:33.152 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.1 0.00 0.00 1 0.001 0 0 2024-01-13T08:58:33.152 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.1 0.00 0.00 1 0.001 0 0 2024-01-13T08:58:33.152 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.1 0.00 0.00 1 0.001 0 0 2024-01-13T08:58:33.152 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: 2024-01-13T08:58:33.152 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: ** Compaction Stats [default] ** 2024-01-13T08:58:33.152 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T08:58:33.152 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T08:58:33.152 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.1 0.00 0.00 1 0.001 0 0 2024-01-13T08:58:33.152 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T08:58:33.152 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-13T08:58:33.152 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-13T08:58:33.153 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: AddFile(Total Files): cumulative 0, interval 0 2024-01-13T08:58:33.153 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-13T08:58:33.153 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: AddFile(Keys): cumulative 0, interval 0 2024-01-13T08:58:33.153 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Cumulative compaction: 0.00 GB write, 0.39 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T08:58:33.153 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Interval compaction: 0.00 GB write, 0.39 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T08:58:33.153 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-13T08:58:33.153 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: 2024-01-13T08:58:33.153 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: ** File Read Latency Histogram By Level [default] ** 2024-01-13T08:58:33.153 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: 2024-01-13T08:58:33.153 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: ** Compaction Stats [default] ** 2024-01-13T08:58:33.153 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T08:58:33.153 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T08:58:33.153 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.1 0.00 0.00 1 0.001 0 0 2024-01-13T08:58:33.153 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.1 0.00 0.00 1 0.001 0 0 2024-01-13T08:58:33.153 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-13T08:58:33.154 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: 2024-01-13T08:58:33.154 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: ** Compaction Stats [default] ** 2024-01-13T08:58:33.154 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T08:58:33.154 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T08:58:33.154 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.1 0.00 0.00 1 0.001 0 0 2024-01-13T08:58:33.154 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T08:58:33.154 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-13T08:58:33.154 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-13T08:58:33.154 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: AddFile(Total Files): cumulative 0, interval 0 2024-01-13T08:58:33.154 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-13T08:58:33.154 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: AddFile(Keys): cumulative 0, interval 0 2024-01-13T08:58:33.155 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Cumulative compaction: 0.00 GB write, 0.39 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T08:58:33.155 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T08:58:33.155 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-13T08:58:33.155 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: 2024-01-13T08:58:33.155 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: ** File Read Latency Histogram By Level [default] ** 2024-01-13T08:58:33.155 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: 2024-01-13T08:58:33.155 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.889+0000 7f6343dbc880 0 mon.smithi141 does not exist in monmap, will attempt to join an existing cluster 2024-01-13T08:58:33.155 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.889+0000 7f6343dbc880 0 using public_addr v2:172.21.15.141:0/0 -> [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] 2024-01-13T08:58:33.155 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.890+0000 7f6343dbc880 0 starting mon.smithi141 rank -1 at public addrs [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] at bind addrs [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi141 fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 2024-01-13T08:58:33.155 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.892+0000 7f6343dbc880 1 mon.smithi141@-1(???) e0 preinit fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 2024-01-13T08:58:33.155 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.918+0000 7f632d3b5700 0 mon.smithi141@-1(synchronizing).mds e1 new map 2024-01-13T08:58:33.155 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.918+0000 7f632d3b5700 0 mon.smithi141@-1(synchronizing).mds e1 print_map 2024-01-13T08:58:33.155 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: e1 2024-01-13T08:58:33.155 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: enable_multiple, ever_enabled_multiple: 1,1 2024-01-13T08:58:33.156 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-13T08:58:33.156 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: legacy client fscid: -1 2024-01-13T08:58:33.156 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: 2024-01-13T08:58:33.156 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: No filesystems configured 2024-01-13T08:58:33.156 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: 2024-01-13T08:58:33.156 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.918+0000 7f632d3b5700 1 mon.smithi141@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-01-13T08:58:33.156 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.918+0000 7f632d3b5700 1 mon.smithi141@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-01-13T08:58:33.156 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.918+0000 7f632d3b5700 1 mon.smithi141@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-01-13T08:58:33.156 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.918+0000 7f632d3b5700 1 mon.smithi141@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-01-13T08:58:33.156 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.918+0000 7f632d3b5700 1 mon.smithi141@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-01-13T08:58:33.157 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.918+0000 7f632d3b5700 1 mon.smithi141@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-01-13T08:58:33.157 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.918+0000 7f632d3b5700 1 mon.smithi141@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-01-13T08:58:33.157 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.918+0000 7f632d3b5700 0 mon.smithi141@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-01-13T08:58:33.157 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.918+0000 7f632d3b5700 0 mon.smithi141@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-01-13T08:58:33.157 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.918+0000 7f632d3b5700 0 mon.smithi141@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-01-13T08:58:33.157 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.918+0000 7f632d3b5700 0 mon.smithi141@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-01-13T08:58:33.157 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:58:30.395957+0000 mgr.smithi077.amvfyi (mgr.14180) 14 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:58:33.157 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:30.447419+0000 mon.smithi077 (mon.0) 167 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:33.157 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:30.449448+0000 mon.smithi077 (mon.0) 168 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-13T08:58:33.157 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:30.450194+0000 mon.smithi077 (mon.0) 169 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:58:33.157 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T08:58:30.450841+0000 mgr.smithi077.amvfyi (mgr.14180) 15 : cephadm [INF] Deploying daemon mon.smithi141 on smithi141 2024-01-13T08:58:33.158 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:31.178587+0000 mon.smithi077 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.141:0/2233958120' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T08:58:33.158 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T08:58:32.919+0000 7f632d3b5700 1 mon.smithi141@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-01-13T08:58:38.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:38 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:32.396397+0000 mgr.smithi077.amvfyi (mgr.14180) 16 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:58:38.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:38 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:32.926273+0000 mon.smithi077 (mon.0) 172 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-01-13T08:58:38.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:38 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:32.926435+0000 mon.smithi077 (mon.0) 173 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mon metadata", "id": "smithi141"}]: dispatch 2024-01-13T08:58:38.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:38 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:32.926566+0000 mon.smithi077 (mon.0) 174 : cluster [INF] mon.smithi077 calling monitor election 2024-01-13T08:58:38.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:38 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:33.923072+0000 mon.smithi077 (mon.0) 175 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mon metadata", "id": "smithi141"}]: dispatch 2024-01-13T08:58:38.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:38 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:34.396699+0000 mgr.smithi077.amvfyi (mgr.14180) 17 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:58:38.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:38 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:34.923169+0000 mon.smithi077 (mon.0) 176 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mon metadata", "id": "smithi141"}]: dispatch 2024-01-13T08:58:38.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:38 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:34.924351+0000 mon.smithi141 (mon.1) 1 : cluster [INF] mon.smithi141 calling monitor election 2024-01-13T08:58:38.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:38 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:35.923131+0000 mon.smithi077 (mon.0) 177 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mon metadata", "id": "smithi141"}]: dispatch 2024-01-13T08:58:38.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:38 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:35.976084+0000 mon.smithi077 (mon.0) 178 : audit [DBG] from='mgr.? 172.21.15.141:0/1163377513' entity='mgr.smithi141.ujrxla' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi141.ujrxla/crt"}]: dispatch 2024-01-13T08:58:38.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:38 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:36.397075+0000 mgr.smithi077.amvfyi (mgr.14180) 18 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:58:38.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:38 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:36.923692+0000 mon.smithi077 (mon.0) 179 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mon metadata", "id": "smithi141"}]: dispatch 2024-01-13T08:58:38.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:38 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:37.923967+0000 mon.smithi077 (mon.0) 180 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mon metadata", "id": "smithi141"}]: dispatch 2024-01-13T08:58:38.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:38 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:37.956712+0000 mon.smithi077 (mon.0) 181 : cluster [INF] mon.smithi077 is new leader, mons smithi077,smithi141 in quorum (ranks 0,1) 2024-01-13T08:58:38.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:38 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:37.961719+0000 mon.smithi077 (mon.0) 182 : cluster [DBG] monmap e2: 2 mons at {smithi077=[v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0],smithi141=[v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0]} removed_ranks: {} 2024-01-13T08:58:38.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:38 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:37.973678+0000 mon.smithi077 (mon.0) 183 : cluster [DBG] fsmap 2024-01-13T08:58:38.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:38 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:37.973742+0000 mon.smithi077 (mon.0) 184 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-01-13T08:58:38.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:38 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:37.973971+0000 mon.smithi077 (mon.0) 185 : cluster [DBG] mgrmap e17: smithi077.amvfyi(active, since 29s) 2024-01-13T08:58:38.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:38 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:37.975289+0000 mon.smithi077 (mon.0) 186 : cluster [DBG] Standby manager daemon smithi141.ujrxla started 2024-01-13T08:58:38.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:38 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:37.977770+0000 mon.smithi077 (mon.0) 187 : cluster [INF] overall HEALTH_OK 2024-01-13T08:58:38.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:38 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:37.982815+0000 mon.smithi077 (mon.0) 188 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:38.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:38 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:37.986086+0000 mon.smithi077 (mon.0) 189 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:38.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:38 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:37.986420+0000 mon.smithi077 (mon.0) 190 : audit [DBG] from='mgr.? 172.21.15.141:0/1163377513' entity='mgr.smithi141.ujrxla' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-01-13T08:58:38.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:38 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:37.988002+0000 mon.smithi077 (mon.0) 191 : audit [DBG] from='mgr.? 172.21.15.141:0/1163377513' entity='mgr.smithi141.ujrxla' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi141.ujrxla/key"}]: dispatch 2024-01-13T08:58:38.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:38 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:37.988473+0000 mon.smithi077 (mon.0) 192 : audit [DBG] from='mgr.? 172.21.15.141:0/1163377513' entity='mgr.smithi141.ujrxla' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-01-13T08:58:38.376 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:38 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:58:32.396397+0000 mgr.smithi077.amvfyi (mgr.14180) 16 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:58:38.376 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:38 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:32.926273+0000 mon.smithi077 (mon.0) 172 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-01-13T08:58:38.376 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:38 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:32.926435+0000 mon.smithi077 (mon.0) 173 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mon metadata", "id": "smithi141"}]: dispatch 2024-01-13T08:58:38.376 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:38 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:58:32.926566+0000 mon.smithi077 (mon.0) 174 : cluster [INF] mon.smithi077 calling monitor election 2024-01-13T08:58:38.376 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:38 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:33.923072+0000 mon.smithi077 (mon.0) 175 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mon metadata", "id": "smithi141"}]: dispatch 2024-01-13T08:58:38.377 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:38 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:58:34.396699+0000 mgr.smithi077.amvfyi (mgr.14180) 17 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:58:38.377 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:38 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:34.923169+0000 mon.smithi077 (mon.0) 176 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mon metadata", "id": "smithi141"}]: dispatch 2024-01-13T08:58:38.377 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:38 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:58:34.924351+0000 mon.smithi141 (mon.1) 1 : cluster [INF] mon.smithi141 calling monitor election 2024-01-13T08:58:38.377 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:38 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:35.923131+0000 mon.smithi077 (mon.0) 177 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mon metadata", "id": "smithi141"}]: dispatch 2024-01-13T08:58:38.377 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:38 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:35.976084+0000 mon.smithi077 (mon.0) 178 : audit [DBG] from='mgr.? 172.21.15.141:0/1163377513' entity='mgr.smithi141.ujrxla' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi141.ujrxla/crt"}]: dispatch 2024-01-13T08:58:38.377 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:38 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:58:36.397075+0000 mgr.smithi077.amvfyi (mgr.14180) 18 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:58:38.377 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:38 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:36.923692+0000 mon.smithi077 (mon.0) 179 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mon metadata", "id": "smithi141"}]: dispatch 2024-01-13T08:58:38.377 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:38 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:37.923967+0000 mon.smithi077 (mon.0) 180 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mon metadata", "id": "smithi141"}]: dispatch 2024-01-13T08:58:38.377 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:38 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:58:37.956712+0000 mon.smithi077 (mon.0) 181 : cluster [INF] mon.smithi077 is new leader, mons smithi077,smithi141 in quorum (ranks 0,1) 2024-01-13T08:58:38.377 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:38 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:58:37.961719+0000 mon.smithi077 (mon.0) 182 : cluster [DBG] monmap e2: 2 mons at {smithi077=[v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0],smithi141=[v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0]} removed_ranks: {} 2024-01-13T08:58:38.377 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:38 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:58:37.973678+0000 mon.smithi077 (mon.0) 183 : cluster [DBG] fsmap 2024-01-13T08:58:38.377 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:38 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:58:37.973742+0000 mon.smithi077 (mon.0) 184 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-01-13T08:58:38.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:38 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:58:37.973971+0000 mon.smithi077 (mon.0) 185 : cluster [DBG] mgrmap e17: smithi077.amvfyi(active, since 29s) 2024-01-13T08:58:38.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:38 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:58:37.975289+0000 mon.smithi077 (mon.0) 186 : cluster [DBG] Standby manager daemon smithi141.ujrxla started 2024-01-13T08:58:38.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:38 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:58:37.977770+0000 mon.smithi077 (mon.0) 187 : cluster [INF] overall HEALTH_OK 2024-01-13T08:58:38.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:38 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:37.982815+0000 mon.smithi077 (mon.0) 188 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:38.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:38 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:37.986086+0000 mon.smithi077 (mon.0) 189 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:38.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:38 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:37.986420+0000 mon.smithi077 (mon.0) 190 : audit [DBG] from='mgr.? 172.21.15.141:0/1163377513' entity='mgr.smithi141.ujrxla' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-01-13T08:58:38.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:38 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:37.988002+0000 mon.smithi077 (mon.0) 191 : audit [DBG] from='mgr.? 172.21.15.141:0/1163377513' entity='mgr.smithi141.ujrxla' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi141.ujrxla/key"}]: dispatch 2024-01-13T08:58:38.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:38 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:37.988473+0000 mon.smithi077 (mon.0) 192 : audit [DBG] from='mgr.? 172.21.15.141:0/1163377513' entity='mgr.smithi141.ujrxla' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-01-13T08:58:38.599 INFO:teuthology.orchestra.run.smithi141.stdout: 2024-01-13T08:58:38.599 INFO:teuthology.orchestra.run.smithi141.stdout:{"epoch":2,"fsid":"99e41a38-b1f1-11ee-95ac-87774f69a715","modified":"2024-01-13T08:58:32.923513Z","created":"2024-01-13T08:56:41.569521Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi141","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:3300","nonce":0},{"type":"v1","addr":"172.21.15.141:6789","nonce":0}]},"addr":"172.21.15.141:6789/0","public_addr":"172.21.15.141:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-01-13T08:58:38.600 INFO:teuthology.orchestra.run.smithi141.stderr:dumped monmap epoch 2 2024-01-13T08:58:39.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:39 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:58:37.984717+0000 mgr.smithi077.amvfyi (mgr.14180) 19 : cephadm [INF] Deploying daemon node-exporter.smithi141 on smithi141 2024-01-13T08:58:39.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:39 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:38.036082+0000 mon.smithi077 (mon.0) 193 : cluster [DBG] mgrmap e18: smithi077.amvfyi(active, since 29s), standbys: smithi141.ujrxla 2024-01-13T08:58:39.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:39 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:38.036247+0000 mon.smithi077 (mon.0) 194 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mgr metadata", "who": "smithi141.ujrxla", "id": "smithi141.ujrxla"}]: dispatch 2024-01-13T08:58:39.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:39 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:38.604525+0000 mon.smithi077 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.141:0/526381476' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T08:58:39.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:39 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:38.923975+0000 mon.smithi077 (mon.0) 196 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mon metadata", "id": "smithi141"}]: dispatch 2024-01-13T08:58:39.356 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:39 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T08:58:37.984717+0000 mgr.smithi077.amvfyi (mgr.14180) 19 : cephadm [INF] Deploying daemon node-exporter.smithi141 on smithi141 2024-01-13T08:58:39.356 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:39 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:58:38.036082+0000 mon.smithi077 (mon.0) 193 : cluster [DBG] mgrmap e18: smithi077.amvfyi(active, since 29s), standbys: smithi141.ujrxla 2024-01-13T08:58:39.356 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:39 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:38.036247+0000 mon.smithi077 (mon.0) 194 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mgr metadata", "who": "smithi141.ujrxla", "id": "smithi141.ujrxla"}]: dispatch 2024-01-13T08:58:39.356 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:39 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:38.604525+0000 mon.smithi077 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.141:0/526381476' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T08:58:39.356 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:39 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:38.923975+0000 mon.smithi077 (mon.0) 196 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mon metadata", "id": "smithi141"}]: dispatch 2024-01-13T08:58:40.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:38.397366+0000 mgr.smithi077.amvfyi (mgr.14180) 20 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:58:40.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:58:38.397366+0000 mgr.smithi077.amvfyi (mgr.14180) 20 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:58:41.103 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-01-13T08:58:41.103 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph config generate-minimal-conf 2024-01-13T08:58:42.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:42 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:40.397809+0000 mgr.smithi077.amvfyi (mgr.14180) 21 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:58:42.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:42 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:58:40.397809+0000 mgr.smithi077.amvfyi (mgr.14180) 21 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:58:42.763 INFO:teuthology.orchestra.run.smithi077.stdout:# minimal ceph.conf for 99e41a38-b1f1-11ee-95ac-87774f69a715 2024-01-13T08:58:42.763 INFO:teuthology.orchestra.run.smithi077.stdout:[global] 2024-01-13T08:58:42.763 INFO:teuthology.orchestra.run.smithi077.stdout: fsid = 99e41a38-b1f1-11ee-95ac-87774f69a715 2024-01-13T08:58:42.763 INFO:teuthology.orchestra.run.smithi077.stdout: mon_host = [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] 2024-01-13T08:58:43.075 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:43 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:42.770755+0000 mon.smithi077 (mon.0) 197 : audit [DBG] from='client.? 172.21.15.77:0/649455878' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:58:43.132 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-01-13T08:58:43.132 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-13T08:58:43.133 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/ceph.conf 2024-01-13T08:58:43.163 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-13T08:58:43.163 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-13T08:58:43.230 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-13T08:58:43.230 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd of=/etc/ceph/ceph.conf 2024-01-13T08:58:43.322 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:43 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:42.770755+0000 mon.smithi077 (mon.0) 197 : audit [DBG] from='client.? 172.21.15.77:0/649455878' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:58:43.323 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-13T08:58:43.323 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-13T08:58:43.418 INFO:tasks.cephadm:Deploying OSDs... 2024-01-13T08:58:43.418 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-13T08:58:43.418 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/scratch_devs of=/dev/stdout 2024-01-13T08:58:43.436 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-01-13T08:58:43.436 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/nvme1n1 2024-01-13T08:58:43.494 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/nvme1n1 2024-01-13T08:58:43.494 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-13T08:58:43.494 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 6h/6d Inode: 191645 Links: 1 Device type: 103,1 2024-01-13T08:58:43.494 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-13T08:58:43.494 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:nvme_device_t:s0 2024-01-13T08:58:43.494 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-01-13 08:57:39.446979860 +0000 2024-01-13T08:58:43.494 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-01-13 08:56:09.818953187 +0000 2024-01-13T08:58:43.495 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-01-13 08:56:09.818953187 +0000 2024-01-13T08:58:43.495 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2024-01-13T08:58:43.495 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-01-13T08:58:43.560 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-01-13T08:58:43.560 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-01-13T08:58:43.560 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000260727 s, 2.0 MB/s 2024-01-13T08:58:43.561 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-01-13T08:58:43.619 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/nvme2n1 2024-01-13T08:58:43.676 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/nvme2n1 2024-01-13T08:58:43.676 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-13T08:58:43.676 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 6h/6d Inode: 191744 Links: 1 Device type: 103,2 2024-01-13T08:58:43.676 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-13T08:58:43.676 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:nvme_device_t:s0 2024-01-13T08:58:43.677 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-01-13 08:57:39.459979573 +0000 2024-01-13T08:58:43.677 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-01-13 08:56:09.970949837 +0000 2024-01-13T08:58:43.677 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-01-13 08:56:09.970949837 +0000 2024-01-13T08:58:43.677 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2024-01-13T08:58:43.677 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-01-13T08:58:43.742 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-01-13T08:58:43.742 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-01-13T08:58:43.742 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000262026 s, 2.0 MB/s 2024-01-13T08:58:43.743 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-01-13T08:58:43.801 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/nvme3n1 2024-01-13T08:58:43.858 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/nvme3n1 2024-01-13T08:58:43.858 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-13T08:58:43.859 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 6h/6d Inode: 191812 Links: 1 Device type: 103,3 2024-01-13T08:58:43.859 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-13T08:58:43.859 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:nvme_device_t:s0 2024-01-13T08:58:43.859 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-01-13 08:57:39.470979331 +0000 2024-01-13T08:58:43.859 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-01-13 08:56:10.120946530 +0000 2024-01-13T08:58:43.859 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-01-13 08:56:10.120946530 +0000 2024-01-13T08:58:43.859 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2024-01-13T08:58:43.859 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-01-13T08:58:43.923 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-01-13T08:58:43.923 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-01-13T08:58:43.924 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000273334 s, 1.9 MB/s 2024-01-13T08:58:43.925 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-01-13T08:58:43.984 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/nvme4n1 2024-01-13T08:58:44.041 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/nvme4n1 2024-01-13T08:58:44.041 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-13T08:58:44.042 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 6h/6d Inode: 196132 Links: 1 Device type: 103,4 2024-01-13T08:58:44.042 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-13T08:58:44.042 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:nvme_device_t:s0 2024-01-13T08:58:44.042 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-01-13 08:57:39.483979044 +0000 2024-01-13T08:58:44.042 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-01-13 08:56:10.275943113 +0000 2024-01-13T08:58:44.042 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-01-13 08:56:10.275943113 +0000 2024-01-13T08:58:44.042 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2024-01-13T08:58:44.042 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-01-13T08:58:44.106 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-01-13T08:58:44.107 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-01-13T08:58:44.107 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000256265 s, 2.0 MB/s 2024-01-13T08:58:44.108 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-01-13T08:58:44.166 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-13T08:58:44.166 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/scratch_devs of=/dev/stdout 2024-01-13T08:58:44.187 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-01-13T08:58:44.187 DEBUG:teuthology.orchestra.run.smithi141:> stat /dev/nvme1n1 2024-01-13T08:58:44.248 INFO:teuthology.orchestra.run.smithi141.stdout: File: /dev/nvme1n1 2024-01-13T08:58:44.248 INFO:teuthology.orchestra.run.smithi141.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-13T08:58:44.248 INFO:teuthology.orchestra.run.smithi141.stdout:Device: 6h/6d Inode: 189958 Links: 1 Device type: 103,1 2024-01-13T08:58:44.248 INFO:teuthology.orchestra.run.smithi141.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-13T08:58:44.248 INFO:teuthology.orchestra.run.smithi141.stdout:Context: system_u:object_r:nvme_device_t:s0 2024-01-13T08:58:44.248 INFO:teuthology.orchestra.run.smithi141.stdout:Access: 2024-01-13 08:58:22.617349834 +0000 2024-01-13T08:58:44.248 INFO:teuthology.orchestra.run.smithi141.stdout:Modify: 2024-01-13 08:56:11.508070140 +0000 2024-01-13T08:58:44.248 INFO:teuthology.orchestra.run.smithi141.stdout:Change: 2024-01-13 08:56:11.508070140 +0000 2024-01-13T08:58:44.248 INFO:teuthology.orchestra.run.smithi141.stdout: Birth: - 2024-01-13T08:58:44.248 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-01-13T08:58:44.315 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:44 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:58:42.398179+0000 mgr.smithi077.amvfyi (mgr.14180) 22 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:58:44.317 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records in 2024-01-13T08:58:44.317 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records out 2024-01-13T08:58:44.317 INFO:teuthology.orchestra.run.smithi141.stderr:512 bytes copied, 0.000310894 s, 1.6 MB/s 2024-01-13T08:58:44.319 DEBUG:teuthology.orchestra.run.smithi141:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-01-13T08:58:44.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:44 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:42.398179+0000 mgr.smithi077.amvfyi (mgr.14180) 22 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:58:44.379 DEBUG:teuthology.orchestra.run.smithi141:> stat /dev/nvme2n1 2024-01-13T08:58:44.437 INFO:teuthology.orchestra.run.smithi141.stdout: File: /dev/nvme2n1 2024-01-13T08:58:44.438 INFO:teuthology.orchestra.run.smithi141.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-13T08:58:44.438 INFO:teuthology.orchestra.run.smithi141.stdout:Device: 6h/6d Inode: 190047 Links: 1 Device type: 103,2 2024-01-13T08:58:44.438 INFO:teuthology.orchestra.run.smithi141.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-13T08:58:44.438 INFO:teuthology.orchestra.run.smithi141.stdout:Context: system_u:object_r:nvme_device_t:s0 2024-01-13T08:58:44.438 INFO:teuthology.orchestra.run.smithi141.stdout:Access: 2024-01-13 08:58:22.629349586 +0000 2024-01-13T08:58:44.438 INFO:teuthology.orchestra.run.smithi141.stdout:Modify: 2024-01-13 08:56:11.660066983 +0000 2024-01-13T08:58:44.438 INFO:teuthology.orchestra.run.smithi141.stdout:Change: 2024-01-13 08:56:11.660066983 +0000 2024-01-13T08:58:44.438 INFO:teuthology.orchestra.run.smithi141.stdout: Birth: - 2024-01-13T08:58:44.438 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-01-13T08:58:44.504 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records in 2024-01-13T08:58:44.505 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records out 2024-01-13T08:58:44.505 INFO:teuthology.orchestra.run.smithi141.stderr:512 bytes copied, 0.000224094 s, 2.3 MB/s 2024-01-13T08:58:44.506 DEBUG:teuthology.orchestra.run.smithi141:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-01-13T08:58:44.564 DEBUG:teuthology.orchestra.run.smithi141:> stat /dev/nvme3n1 2024-01-13T08:58:44.622 INFO:teuthology.orchestra.run.smithi141.stdout: File: /dev/nvme3n1 2024-01-13T08:58:44.622 INFO:teuthology.orchestra.run.smithi141.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-13T08:58:44.622 INFO:teuthology.orchestra.run.smithi141.stdout:Device: 6h/6d Inode: 190126 Links: 1 Device type: 103,3 2024-01-13T08:58:44.622 INFO:teuthology.orchestra.run.smithi141.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-13T08:58:44.622 INFO:teuthology.orchestra.run.smithi141.stdout:Context: system_u:object_r:nvme_device_t:s0 2024-01-13T08:58:44.622 INFO:teuthology.orchestra.run.smithi141.stdout:Access: 2024-01-13 08:58:22.641349337 +0000 2024-01-13T08:58:44.622 INFO:teuthology.orchestra.run.smithi141.stdout:Modify: 2024-01-13 08:56:11.809063889 +0000 2024-01-13T08:58:44.622 INFO:teuthology.orchestra.run.smithi141.stdout:Change: 2024-01-13 08:56:11.809063889 +0000 2024-01-13T08:58:44.622 INFO:teuthology.orchestra.run.smithi141.stdout: Birth: - 2024-01-13T08:58:44.623 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-01-13T08:58:44.688 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records in 2024-01-13T08:58:44.688 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records out 2024-01-13T08:58:44.688 INFO:teuthology.orchestra.run.smithi141.stderr:512 bytes copied, 0.000211194 s, 2.4 MB/s 2024-01-13T08:58:44.690 DEBUG:teuthology.orchestra.run.smithi141:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-01-13T08:58:44.748 DEBUG:teuthology.orchestra.run.smithi141:> stat /dev/nvme4n1 2024-01-13T08:58:44.805 INFO:teuthology.orchestra.run.smithi141.stdout: File: /dev/nvme4n1 2024-01-13T08:58:44.805 INFO:teuthology.orchestra.run.smithi141.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-13T08:58:44.806 INFO:teuthology.orchestra.run.smithi141.stdout:Device: 6h/6d Inode: 187139 Links: 1 Device type: 103,4 2024-01-13T08:58:44.806 INFO:teuthology.orchestra.run.smithi141.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-13T08:58:44.806 INFO:teuthology.orchestra.run.smithi141.stdout:Context: system_u:object_r:nvme_device_t:s0 2024-01-13T08:58:44.806 INFO:teuthology.orchestra.run.smithi141.stdout:Access: 2024-01-13 08:58:22.652349109 +0000 2024-01-13T08:58:44.806 INFO:teuthology.orchestra.run.smithi141.stdout:Modify: 2024-01-13 08:56:11.960060753 +0000 2024-01-13T08:58:44.806 INFO:teuthology.orchestra.run.smithi141.stdout:Change: 2024-01-13 08:56:11.960060753 +0000 2024-01-13T08:58:44.806 INFO:teuthology.orchestra.run.smithi141.stdout: Birth: - 2024-01-13T08:58:44.806 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-01-13T08:58:44.871 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records in 2024-01-13T08:58:44.871 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records out 2024-01-13T08:58:44.871 INFO:teuthology.orchestra.run.smithi141.stderr:512 bytes copied, 0.000244432 s, 2.1 MB/s 2024-01-13T08:58:44.872 DEBUG:teuthology.orchestra.run.smithi141:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-01-13T08:58:44.929 DEBUG:teuthology.orchestra.run.smithi141:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph orch apply osd --all-available-devices 2024-01-13T08:58:46.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:44.398400+0000 mgr.smithi077.amvfyi (mgr.14180) 23 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:58:46.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:46 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:58:44.398400+0000 mgr.smithi077.amvfyi (mgr.14180) 23 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:58:48.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:48 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:46.398633+0000 mgr.smithi077.amvfyi (mgr.14180) 24 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:58:48.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:48 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:47.595938+0000 mon.smithi077 (mon.0) 198 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:48.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:48 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:47.597607+0000 mon.smithi077 (mon.0) 199 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:58:48.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:48 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:47.991552+0000 mon.smithi077 (mon.0) 200 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:48.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:48 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:58:46.398633+0000 mgr.smithi077.amvfyi (mgr.14180) 24 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:58:48.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:48 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:47.595938+0000 mon.smithi077 (mon.0) 198 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:48.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:48 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:47.597607+0000 mon.smithi077 (mon.0) 199 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:58:48.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:48 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:47.991552+0000 mon.smithi077 (mon.0) 200 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:48.478 INFO:teuthology.orchestra.run.smithi141.stdout:Scheduled osd.all-available-devices update... 2024-01-13T08:58:48.950 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-01-13T08:58:48.951 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:58:49.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:49 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:48.398908+0000 mgr.smithi077.amvfyi (mgr.14180) 25 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:58:49.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:49 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:48.479348+0000 mgr.smithi077.amvfyi (mgr.14180) 26 : audit [DBG] from='client.24101 -' entity='client.admin' cmd=[{"prefix": "orch apply osd", "all_available_devices": true, "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T08:58:49.577 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:49 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:58:48.481063+0000 mgr.smithi077.amvfyi (mgr.14180) 27 : cephadm [INF] Marking host: smithi077 for OSDSpec preview refresh. 2024-01-13T08:58:49.577 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:49 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:58:48.481125+0000 mgr.smithi077.amvfyi (mgr.14180) 28 : cephadm [INF] Marking host: smithi141 for OSDSpec preview refresh. 2024-01-13T08:58:49.577 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:49 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:58:48.481420+0000 mgr.smithi077.amvfyi (mgr.14180) 29 : cephadm [INF] Saving service osd.all-available-devices spec with placement * 2024-01-13T08:58:49.577 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:49 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:48.485144+0000 mon.smithi077 (mon.0) 201 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:49.577 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:49 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:49.283305+0000 mon.smithi077 (mon.0) 202 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:49.577 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:49 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:49.289960+0000 mon.smithi077 (mon.0) 203 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:49.578 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:49 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:49.290758+0000 mon.smithi077 (mon.0) 204 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:58:49.578 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:49 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:49.291419+0000 mon.smithi077 (mon.0) 205 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T08:58:49.578 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:49 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:49.318535+0000 mon.smithi077 (mon.0) 206 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:49.578 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:49 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:49.321513+0000 mon.smithi077 (mon.0) 207 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:49.578 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:49 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:49.324707+0000 mon.smithi077 (mon.0) 208 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:49.578 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:49 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:49.329331+0000 mon.smithi077 (mon.0) 209 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T08:58:49.578 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:49 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:49.332457+0000 mon.smithi077 (mon.0) 210 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:58:49.578 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:49 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:49.333405+0000 mon.smithi077 (mon.0) 211 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:58:49.578 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:49 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:49.333563+0000 mon.smithi077 (mon.0) 212 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:58:49.579 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:49 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:49.333961+0000 mon.smithi077 (mon.0) 213 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:58:49.751 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:49 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:58:48.398908+0000 mgr.smithi077.amvfyi (mgr.14180) 25 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:58:49.751 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:49 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:48.479348+0000 mgr.smithi077.amvfyi (mgr.14180) 26 : audit [DBG] from='client.24101 -' entity='client.admin' cmd=[{"prefix": "orch apply osd", "all_available_devices": true, "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T08:58:49.751 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:49 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T08:58:48.481063+0000 mgr.smithi077.amvfyi (mgr.14180) 27 : cephadm [INF] Marking host: smithi077 for OSDSpec preview refresh. 2024-01-13T08:58:49.751 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:49 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T08:58:48.481125+0000 mgr.smithi077.amvfyi (mgr.14180) 28 : cephadm [INF] Marking host: smithi141 for OSDSpec preview refresh. 2024-01-13T08:58:49.751 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:49 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T08:58:48.481420+0000 mgr.smithi077.amvfyi (mgr.14180) 29 : cephadm [INF] Saving service osd.all-available-devices spec with placement * 2024-01-13T08:58:49.751 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:49 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:48.485144+0000 mon.smithi077 (mon.0) 201 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:49.751 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:49 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:49.283305+0000 mon.smithi077 (mon.0) 202 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:49.751 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:49 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:49.289960+0000 mon.smithi077 (mon.0) 203 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:49.751 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:49 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:49.290758+0000 mon.smithi077 (mon.0) 204 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:58:49.752 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:49 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:49.291419+0000 mon.smithi077 (mon.0) 205 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T08:58:49.752 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:49 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:49.318535+0000 mon.smithi077 (mon.0) 206 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:49.752 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:49 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:49.321513+0000 mon.smithi077 (mon.0) 207 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:49.752 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:49 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:49.324707+0000 mon.smithi077 (mon.0) 208 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:49.752 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:49 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:49.329331+0000 mon.smithi077 (mon.0) 209 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T08:58:49.752 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:49 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:49.332457+0000 mon.smithi077 (mon.0) 210 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:58:49.752 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:49 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:49.333405+0000 mon.smithi077 (mon.0) 211 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:58:49.752 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:49 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:49.333563+0000 mon.smithi077 (mon.0) 212 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:58:49.752 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:49 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:49.333961+0000 mon.smithi077 (mon.0) 213 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:58:50.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:50 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T08:58:49.292320+0000 mgr.smithi077.amvfyi (mgr.14180) 30 : cephadm [INF] Updating smithi077:/etc/ceph/ceph.conf 2024-01-13T08:58:50.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:50 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T08:58:49.292524+0000 mgr.smithi077.amvfyi (mgr.14180) 31 : cephadm [INF] Updating smithi141:/etc/ceph/ceph.conf 2024-01-13T08:58:50.709 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:50 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:58:49.292320+0000 mgr.smithi077.amvfyi (mgr.14180) 30 : cephadm [INF] Updating smithi077:/etc/ceph/ceph.conf 2024-01-13T08:58:50.709 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:50 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:58:49.292524+0000 mgr.smithi077.amvfyi (mgr.14180) 31 : cephadm [INF] Updating smithi141:/etc/ceph/ceph.conf 2024-01-13T08:58:50.839 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T08:58:51.212 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2024-01-13T08:58:51.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:51 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:50.399382+0000 mgr.smithi077.amvfyi (mgr.14180) 32 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:58:51.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:51 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:50.846723+0000 mon.smithi077 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.77:0/4081034955' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:58:51.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:51 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:51.334659+0000 mon.smithi141 (mon.1) 2 : audit [INF] from='client.? 172.21.15.141:0/3161831228' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "342496a5-f32d-4df2-ace1-b5f9540733bb"}]: dispatch 2024-01-13T08:58:51.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:51 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:51.335046+0000 mon.smithi077 (mon.0) 215 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "342496a5-f32d-4df2-ace1-b5f9540733bb"}]: dispatch 2024-01-13T08:58:51.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:51 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:51.348457+0000 mon.smithi077 (mon.0) 216 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "342496a5-f32d-4df2-ace1-b5f9540733bb"}]': finished 2024-01-13T08:58:51.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:51 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:51.348506+0000 mon.smithi077 (mon.0) 217 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-01-13T08:58:51.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:51 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:51.348605+0000 mon.smithi077 (mon.0) 218 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:58:51.867 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:51 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:58:50.399382+0000 mgr.smithi077.amvfyi (mgr.14180) 32 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:58:51.867 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:51 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:50.846723+0000 mon.smithi077 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.77:0/4081034955' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:58:51.867 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:51 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:51.334659+0000 mon.smithi141 (mon.1) 2 : audit [INF] from='client.? 172.21.15.141:0/3161831228' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "342496a5-f32d-4df2-ace1-b5f9540733bb"}]: dispatch 2024-01-13T08:58:51.867 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:51 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:51.335046+0000 mon.smithi077 (mon.0) 215 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "342496a5-f32d-4df2-ace1-b5f9540733bb"}]: dispatch 2024-01-13T08:58:51.868 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:51 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:51.348457+0000 mon.smithi077 (mon.0) 216 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "342496a5-f32d-4df2-ace1-b5f9540733bb"}]': finished 2024-01-13T08:58:51.868 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:51 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:58:51.348506+0000 mon.smithi077 (mon.0) 217 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-01-13T08:58:51.868 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:51 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:51.348605+0000 mon.smithi077 (mon.0) 218 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:58:52.214 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:58:52.719 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:51.635546+0000 mon.smithi141 (mon.1) 3 : audit [INF] from='client.? 172.21.15.141:0/2439897709' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:58:52.720 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:51.635910+0000 mon.smithi077 (mon.0) 219 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:58:52.720 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:51.639247+0000 mon.smithi077 (mon.0) 220 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:58:52.720 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:51.639294+0000 mon.smithi077 (mon.0) 221 : cluster [DBG] osdmap e7: 0 total, 0 up, 0 in 2024-01-13T08:58:52.720 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:51.793279+0000 mon.smithi077 (mon.0) 222 : audit [INF] from='client.? 172.21.15.77:0/4054443971' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "578d5d59-1a17-404a-af27-47d5673e5d05"}]: dispatch 2024-01-13T08:58:52.720 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:51.796832+0000 mon.smithi077 (mon.0) 223 : audit [INF] from='client.? 172.21.15.77:0/4054443971' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "578d5d59-1a17-404a-af27-47d5673e5d05"}]': finished 2024-01-13T08:58:52.720 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:51.796868+0000 mon.smithi077 (mon.0) 224 : cluster [DBG] osdmap e8: 1 total, 0 up, 1 in 2024-01-13T08:58:52.720 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:51.796945+0000 mon.smithi077 (mon.0) 225 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:58:52.720 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:52.084359+0000 mon.smithi077 (mon.0) 226 : audit [INF] from='client.? 172.21.15.77:0/2790731839' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:58:52.720 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:58:52.087108+0000 mgr.smithi077.amvfyi (mgr.14180) 33 : cephadm [ERR] executing create_from_spec_one(([('smithi077', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi141', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:58:52.720 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T08:58:52.720 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:58:52.720 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return f(*arg) 2024-01-13T08:58:52.721 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:58:52.721 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:58:52.721 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:58:52.721 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: code, '\n'.join(err))) 2024-01-13T08:58:52.721 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T08:58:52.721 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpesudkjaq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbey7y289:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:58:52.721 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:58:52.721 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:58:52.721 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:58:52.721 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 342496a5-f32d-4df2-ace1-b5f9540733bb 2024-01-13T08:58:52.721 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:58:52.721 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:58:52.721 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:58:52.722 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:58:52.722 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:58:52.722 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.prepare() 2024-01-13T08:58:52.722 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:58:52.722 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:58:52.722 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:58:52.722 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:58:52.722 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:58:52.722 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:58:52.722 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:58:52.722 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:58:52.722 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:58:52.722 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T08:58:52.722 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:58:52.722 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T08:58:52.723 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:58:52.723 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:58:52.723 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:58:52.723 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:58:52.723 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:58:52.723 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:58:52.723 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:58:52.723 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:58:52.723 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:58:52.723 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:58:52.723 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T08:58:52.723 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:58:52.723 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:58:52.723 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:58:52.724 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T08:58:52.724 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:58:52.724 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:58:52.724 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:58:52.724 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:58:52.724 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:58:52.724 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:58:52.724 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:58:52.724 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:58:52.724 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:58:52.724 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:58:52.724 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:58:52.724 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:58:52.724 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:58:52.724 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:58:52.725 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:58:52.725 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:58:52.725 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:58:52.725 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:58:52.725 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:58:52.725 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:58:52.725 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:58:52.725 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:58:52.725 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:58:52.725 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T08:58:52.725 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:58:52.725 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: main() 2024-01-13T08:58:52.725 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:58:52.725 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: r = ctx.func(ctx) 2024-01-13T08:58:52.725 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:58:52.726 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:58:52.726 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:58:52.726 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:58:52.726 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:58:52.726 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:58:52.726 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:58:52.726 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:58:52.726 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:58:52.726 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:58:52.726 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:58:52.726 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:58:52.726 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpesudkjaq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbey7y289:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:58:52.726 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:52.087835+0000 mon.smithi077 (mon.0) 227 : audit [INF] from='client.? 172.21.15.77:0/2790731839' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:58:52.726 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:52.087872+0000 mon.smithi077 (mon.0) 228 : cluster [DBG] osdmap e9: 0 total, 0 up, 0 in 2024-01-13T08:58:52.879 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:51.635546+0000 mon.smithi141 (mon.1) 3 : audit [INF] from='client.? 172.21.15.141:0/2439897709' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:58:52.879 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:51.635910+0000 mon.smithi077 (mon.0) 219 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:58:52.879 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:51.639247+0000 mon.smithi077 (mon.0) 220 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:58:52.880 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:58:51.639294+0000 mon.smithi077 (mon.0) 221 : cluster [DBG] osdmap e7: 0 total, 0 up, 0 in 2024-01-13T08:58:52.880 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:51.793279+0000 mon.smithi077 (mon.0) 222 : audit [INF] from='client.? 172.21.15.77:0/4054443971' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "578d5d59-1a17-404a-af27-47d5673e5d05"}]: dispatch 2024-01-13T08:58:52.880 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:51.796832+0000 mon.smithi077 (mon.0) 223 : audit [INF] from='client.? 172.21.15.77:0/4054443971' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "578d5d59-1a17-404a-af27-47d5673e5d05"}]': finished 2024-01-13T08:58:52.880 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:58:51.796868+0000 mon.smithi077 (mon.0) 224 : cluster [DBG] osdmap e8: 1 total, 0 up, 1 in 2024-01-13T08:58:52.880 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:51.796945+0000 mon.smithi077 (mon.0) 225 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:58:52.880 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:52.084359+0000 mon.smithi077 (mon.0) 226 : audit [INF] from='client.? 172.21.15.77:0/2790731839' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:58:52.880 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T08:58:52.087108+0000 mgr.smithi077.amvfyi (mgr.14180) 33 : cephadm [ERR] executing create_from_spec_one(([('smithi077', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi141', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:58:52.880 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T08:58:52.880 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:58:52.880 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return f(*arg) 2024-01-13T08:58:52.880 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:58:52.880 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:58:52.880 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:58:52.881 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: code, '\n'.join(err))) 2024-01-13T08:58:52.881 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T08:58:52.881 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpesudkjaq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbey7y289:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:58:52.881 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:58:52.881 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:58:52.881 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:58:52.881 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 342496a5-f32d-4df2-ace1-b5f9540733bb 2024-01-13T08:58:52.881 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:58:52.881 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:58:52.881 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:58:52.881 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:58:52.881 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:58:52.881 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.prepare() 2024-01-13T08:58:52.881 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:58:52.882 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:58:52.882 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:58:52.882 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:58:52.882 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:58:52.882 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:58:52.883 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:58:52.883 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:58:52.883 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:58:52.883 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T08:58:52.883 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:58:52.883 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T08:58:52.883 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:58:52.883 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:58:52.883 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:58:52.883 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:58:52.883 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:58:52.883 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:58:52.883 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:58:52.883 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:58:52.883 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:58:52.884 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:58:52.884 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T08:58:52.884 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:58:52.884 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:58:52.884 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:58:52.884 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T08:58:52.884 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:58:52.884 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:58:52.884 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:58:52.884 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:58:52.884 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:58:52.884 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:58:52.884 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:58:52.884 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:58:52.885 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:58:52.885 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:58:52.885 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:58:52.885 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:58:52.885 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:58:52.885 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:58:52.885 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:58:52.885 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:58:52.885 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:58:52.885 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:58:52.885 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:58:52.885 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:58:52.885 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:58:52.885 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:58:52.885 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:58:52.885 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T08:58:52.886 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:58:52.886 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: main() 2024-01-13T08:58:52.886 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:58:52.886 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: r = ctx.func(ctx) 2024-01-13T08:58:52.886 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:58:52.886 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:58:52.886 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:58:52.886 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:58:52.886 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:58:52.886 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:58:52.886 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:58:52.886 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:58:52.886 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:58:52.886 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:58:52.887 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:58:52.887 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:58:52.887 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpesudkjaq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbey7y289:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:58:52.887 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:52.087835+0000 mon.smithi077 (mon.0) 227 : audit [INF] from='client.? 172.21.15.77:0/2790731839' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:58:52.887 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:58:52.087872+0000 mon.smithi077 (mon.0) 228 : cluster [DBG] osdmap e9: 0 total, 0 up, 0 in 2024-01-13T08:58:53.525 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:52.399636+0000 mgr.smithi077.amvfyi (mgr.14180) 34 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:58:53.525 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:58:52.613046+0000 mgr.smithi077.amvfyi (mgr.14180) 35 : cephadm [ERR] executing create_from_spec_one(([('smithi077', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi141', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:58:53.525 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T08:58:53.525 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:58:53.526 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return f(*arg) 2024-01-13T08:58:53.526 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:58:53.526 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:58:53.526 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:58:53.526 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: code, '\n'.join(err))) 2024-01-13T08:58:53.526 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi077/config 2024-01-13T08:58:53.526 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp54jte5ke:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp45r911qs:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:58:53.526 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:58:53.526 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:58:53.526 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:58:53.527 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 578d5d59-1a17-404a-af27-47d5673e5d05 2024-01-13T08:58:53.527 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:58:53.527 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:58:53.527 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:58:53.528 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:58:53.528 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:58:53.528 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.prepare() 2024-01-13T08:58:53.528 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:58:53.528 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:58:53.528 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:58:53.528 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:58:53.528 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:58:53.528 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:58:53.528 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:58:53.528 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:58:53.528 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:58:53.528 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T08:58:53.528 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:58:53.528 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T08:58:53.529 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:58:53.529 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:58:53.529 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:58:53.529 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:58:53.529 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:58:53.529 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:58:53.529 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:58:53.529 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:58:53.529 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:58:53.529 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:58:53.529 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T08:58:53.529 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:58:53.529 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:58:53.529 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:58:53.529 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T08:58:53.530 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:58:53.530 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:58:53.530 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:58:53.530 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:58:53.530 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:58:53.530 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:58:53.530 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:58:53.530 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:58:53.530 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:58:53.531 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:58:53.531 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:58:53.531 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:58:53.531 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:58:53.531 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:58:53.531 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:58:53.531 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:58:53.531 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:58:53.531 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:58:53.531 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:58:53.531 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:58:53.531 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:58:53.531 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:58:53.531 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:58:53.532 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T08:58:53.532 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:58:53.532 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: main() 2024-01-13T08:58:53.532 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:58:53.532 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: r = ctx.func(ctx) 2024-01-13T08:58:53.532 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:58:53.532 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:58:53.532 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:58:53.532 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:58:53.532 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:58:53.532 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:58:53.532 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:58:53.532 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:58:53.532 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:58:53.533 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:58:53.533 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:58:53.533 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:58:53.533 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp54jte5ke:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp45r911qs:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:58:53.533 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:58:52.615494+0000 mgr.smithi077.amvfyi (mgr.14180) 36 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T08:58:53.533 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: service_id: all-available-devices 2024-01-13T08:58:53.533 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: service_name: osd.all-available-devices 2024-01-13T08:58:53.533 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: placement: 2024-01-13T08:58:53.533 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: host_pattern: '*' 2024-01-13T08:58:53.533 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: spec: 2024-01-13T08:58:53.533 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: data_devices: 2024-01-13T08:58:53.533 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: all: true 2024-01-13T08:58:53.533 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: filter_logic: AND 2024-01-13T08:58:53.533 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: objectstore: bluestore 2024-01-13T08:58:53.533 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T08:58:53.534 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpesudkjaq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbey7y289:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:58:53.534 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:58:53.534 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:58:53.534 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:58:53.534 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 342496a5-f32d-4df2-ace1-b5f9540733bb 2024-01-13T08:58:53.534 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:58:53.534 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:58:53.534 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:58:53.534 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:58:53.534 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:58:53.534 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.prepare() 2024-01-13T08:58:53.534 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:58:53.534 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:58:53.534 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:58:53.534 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:58:53.535 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:58:53.535 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:58:53.535 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:58:53.535 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:58:53.535 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:58:53.535 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T08:58:53.535 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:58:53.535 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T08:58:53.535 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:58:53.535 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:58:53.535 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:58:53.535 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:58:53.535 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:58:53.535 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:58:53.535 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:58:53.536 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:58:53.536 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:58:53.536 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:58:53.536 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T08:58:53.536 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:58:53.536 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:58:53.536 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:58:53.536 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T08:58:53.536 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:58:53.536 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:58:53.536 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:58:53.536 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:58:53.536 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:58:53.536 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:58:53.536 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:58:53.538 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:58:53.538 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:58:53.538 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:58:53.538 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:58:53.538 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:58:53.538 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:58:53.538 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:58:53.538 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:58:53.538 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:58:53.538 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:58:53.538 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:58:53.538 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:58:53.539 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:58:53.539 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:58:53.539 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:58:53.539 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:58:53.539 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T08:58:53.539 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:58:53.539 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: main() 2024-01-13T08:58:53.539 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:58:53.539 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: r = ctx.func(ctx) 2024-01-13T08:58:53.539 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:58:53.539 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:58:53.539 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:58:53.539 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:58:53.539 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:58:53.540 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:58:53.540 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:58:53.540 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:58:53.540 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:58:53.540 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:58:53.540 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:58:53.540 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:58:53.540 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpesudkjaq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbey7y289:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:58:53.540 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T08:58:53.541 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T08:58:53.541 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: if self._apply_service(spec): 2024-01-13T08:58:53.541 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T08:58:53.541 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T08:58:53.541 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T08:58:53.541 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T08:58:53.541 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T08:58:53.541 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T08:58:53.541 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T08:58:53.541 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T08:58:53.541 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T08:58:53.541 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: raise self._value 2024-01-13T08:58:53.541 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T08:58:53.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: result = (True, func(*args, **kwds)) 2024-01-13T08:58:53.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T08:58:53.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return list(map(*args)) 2024-01-13T08:58:53.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:58:53.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return f(*arg) 2024-01-13T08:58:53.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:58:53.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:58:53.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:58:53.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: code, '\n'.join(err))) 2024-01-13T08:58:53.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T08:58:53.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpesudkjaq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbey7y289:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:58:53.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:58:53.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:58:53.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:58:53.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 342496a5-f32d-4df2-ace1-b5f9540733bb 2024-01-13T08:58:53.543 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:58:53.543 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:58:53.543 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:58:53.543 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:58:53.543 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:58:53.543 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.prepare() 2024-01-13T08:58:53.543 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:58:53.543 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:58:53.543 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:58:53.543 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:58:53.543 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:58:53.543 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:58:53.543 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:58:53.544 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:58:53.544 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:58:53.544 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T08:58:53.544 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:58:53.544 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T08:58:53.544 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:58:53.544 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:58:53.544 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:58:53.544 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:58:53.544 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:58:53.544 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:58:53.544 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:58:53.545 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:58:53.545 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:58:53.545 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:58:53.545 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T08:58:53.545 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:58:53.545 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:58:53.545 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:58:53.545 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T08:58:53.545 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:58:53.545 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:58:53.545 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:58:53.545 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:58:53.545 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:58:53.545 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:58:53.545 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:58:53.546 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:58:53.546 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:58:53.546 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:58:53.546 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:58:53.546 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:58:53.546 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:58:53.546 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:58:53.546 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:58:53.546 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:58:53.546 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:58:53.546 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:58:53.546 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:58:53.546 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:58:53.546 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:58:53.546 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:58:53.547 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:58:53.547 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T08:58:53.547 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:58:53.547 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: main() 2024-01-13T08:58:53.547 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:58:53.547 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: r = ctx.func(ctx) 2024-01-13T08:58:53.547 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:58:53.547 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:58:53.547 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:58:53.547 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:58:53.547 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:58:53.547 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:58:53.547 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:58:53.547 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:58:53.547 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:58:53.548 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:58:53.548 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:58:53.548 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:58:53.548 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpesudkjaq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbey7y289:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:58:53.548 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:58:52.616263+0000 mgr.smithi077.amvfyi (mgr.14180) 37 : cephadm [INF] Reconfiguring mon.smithi077 (unknown last config time)... 2024-01-13T08:58:53.548 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:52.616279+0000 mgr.smithi077.amvfyi (mgr.14180) 38 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:58:53.548 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:52.616826+0000 mon.smithi077 (mon.0) 229 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-13T08:58:53.548 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:52.617488+0000 mon.smithi077 (mon.0) 230 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-13T08:58:53.548 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:52.618058+0000 mon.smithi077 (mon.0) 231 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:58:53.548 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:58:52.618609+0000 mgr.smithi077.amvfyi (mgr.14180) 39 : cephadm [INF] Reconfiguring daemon mon.smithi077 on smithi077 2024-01-13T08:58:53.548 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:53.086159+0000 mon.smithi077 (mon.0) 232 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T08:58:53.719 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T08:58:53.881 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:58:52.399636+0000 mgr.smithi077.amvfyi (mgr.14180) 34 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:58:53.881 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T08:58:52.613046+0000 mgr.smithi077.amvfyi (mgr.14180) 35 : cephadm [ERR] executing create_from_spec_one(([('smithi077', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi141', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:58:53.882 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T08:58:53.882 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:58:53.882 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return f(*arg) 2024-01-13T08:58:53.882 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:58:53.882 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:58:53.882 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:58:53.882 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: code, '\n'.join(err))) 2024-01-13T08:58:53.882 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi077/config 2024-01-13T08:58:53.882 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp54jte5ke:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp45r911qs:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:58:53.882 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:58:53.882 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:58:53.882 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:58:53.882 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 578d5d59-1a17-404a-af27-47d5673e5d05 2024-01-13T08:58:53.883 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:58:53.883 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:58:53.883 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:58:53.883 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:58:53.883 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:58:53.883 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.prepare() 2024-01-13T08:58:53.883 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:58:53.883 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:58:53.883 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:58:53.883 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:58:53.883 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:58:53.883 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:58:53.883 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:58:53.883 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:58:53.884 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:58:53.884 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T08:58:53.884 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:58:53.884 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T08:58:53.885 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:58:53.885 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:58:53.885 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:58:53.885 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:58:53.885 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:58:53.885 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:58:53.885 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:58:53.885 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:58:53.885 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:58:53.885 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:58:53.885 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T08:58:53.885 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:58:53.886 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:58:53.886 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:58:53.886 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T08:58:53.886 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:58:53.886 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:58:53.886 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:58:53.886 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:58:53.886 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:58:53.886 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:58:53.886 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:58:53.886 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:58:53.886 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:58:53.886 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:58:53.887 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:58:53.887 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:58:53.887 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:58:53.887 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:58:53.887 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:58:53.887 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:58:53.887 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:58:53.887 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:58:53.887 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:58:53.887 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:58:53.887 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:58:53.887 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:58:53.887 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:58:53.887 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T08:58:53.888 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:58:53.888 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: main() 2024-01-13T08:58:53.888 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:58:53.888 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: r = ctx.func(ctx) 2024-01-13T08:58:53.888 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:58:53.888 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:58:53.888 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:58:53.888 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:58:53.888 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:58:53.888 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:58:53.888 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:58:53.888 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:58:53.889 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:58:53.889 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:58:53.889 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:58:53.889 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:58:53.889 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp54jte5ke:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp45r911qs:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:58:53.890 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T08:58:52.615494+0000 mgr.smithi077.amvfyi (mgr.14180) 36 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T08:58:53.890 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: service_id: all-available-devices 2024-01-13T08:58:53.890 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: service_name: osd.all-available-devices 2024-01-13T08:58:53.890 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: placement: 2024-01-13T08:58:53.890 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: host_pattern: '*' 2024-01-13T08:58:53.890 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: spec: 2024-01-13T08:58:53.890 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: data_devices: 2024-01-13T08:58:53.890 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: all: true 2024-01-13T08:58:53.890 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: filter_logic: AND 2024-01-13T08:58:53.890 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: objectstore: bluestore 2024-01-13T08:58:53.890 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T08:58:53.890 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpesudkjaq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbey7y289:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:58:53.891 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:58:53.891 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:58:53.891 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:58:53.891 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 342496a5-f32d-4df2-ace1-b5f9540733bb 2024-01-13T08:58:53.891 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:58:53.891 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:58:53.891 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:58:53.891 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:58:53.891 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:58:53.891 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.prepare() 2024-01-13T08:58:53.891 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:58:53.891 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:58:53.891 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:58:53.891 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:58:53.892 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:58:53.892 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:58:53.892 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:58:53.892 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:58:53.892 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:58:53.892 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T08:58:53.892 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:58:53.892 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T08:58:53.892 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:58:53.892 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:58:53.892 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:58:53.892 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:58:53.892 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:58:53.893 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:58:53.893 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:58:53.893 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:58:53.893 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:58:53.893 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:58:53.893 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T08:58:53.893 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:58:53.893 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:58:53.893 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:58:53.893 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T08:58:53.893 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:58:53.893 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:58:53.893 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:58:53.894 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:58:53.894 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:58:53.894 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:58:53.894 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:58:53.894 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:58:53.894 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:58:53.894 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:58:53.894 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:58:53.895 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:58:53.895 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:58:53.895 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:58:53.895 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:58:53.895 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:58:53.895 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:58:53.895 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:58:53.895 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:58:53.895 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:58:53.895 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:58:53.895 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:58:53.895 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:58:53.896 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T08:58:53.896 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:58:53.896 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: main() 2024-01-13T08:58:53.896 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:58:53.896 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: r = ctx.func(ctx) 2024-01-13T08:58:53.896 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:58:53.896 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:58:53.896 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:58:53.896 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:58:53.897 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:58:53.897 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:58:53.897 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:58:53.897 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:58:53.897 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:58:53.897 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:58:53.897 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:58:53.897 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:58:53.897 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpesudkjaq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbey7y289:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:58:53.897 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T08:58:53.897 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T08:58:53.897 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: if self._apply_service(spec): 2024-01-13T08:58:53.897 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T08:58:53.898 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T08:58:53.898 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T08:58:53.898 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T08:58:53.898 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T08:58:53.898 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T08:58:53.898 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T08:58:53.898 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T08:58:53.898 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T08:58:53.898 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: raise self._value 2024-01-13T08:58:53.898 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T08:58:53.898 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: result = (True, func(*args, **kwds)) 2024-01-13T08:58:53.898 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T08:58:53.898 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return list(map(*args)) 2024-01-13T08:58:53.898 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:58:53.898 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return f(*arg) 2024-01-13T08:58:53.899 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:58:53.899 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:58:53.899 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:58:53.899 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: code, '\n'.join(err))) 2024-01-13T08:58:53.899 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T08:58:53.899 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpesudkjaq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbey7y289:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:58:53.899 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:58:53.900 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:58:53.900 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:58:53.900 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 342496a5-f32d-4df2-ace1-b5f9540733bb 2024-01-13T08:58:53.900 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:58:53.900 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:58:53.900 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:58:53.900 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:58:53.900 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:58:53.900 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.prepare() 2024-01-13T08:58:53.900 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:58:53.900 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:58:53.900 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:58:53.900 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:58:53.901 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:58:53.901 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:58:53.901 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:58:53.901 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:58:53.901 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:58:53.901 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T08:58:53.901 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:58:53.901 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T08:58:53.901 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:58:53.901 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:58:53.901 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:58:53.901 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:58:53.901 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:58:53.901 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:58:53.901 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:58:53.902 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:58:53.902 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:58:53.902 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:58:53.902 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T08:58:53.902 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:58:53.902 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:58:53.902 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:58:53.902 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T08:58:53.902 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:58:53.902 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:58:53.902 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:58:53.902 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:58:53.902 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:58:53.902 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:58:53.903 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:58:53.903 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:58:53.903 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:58:53.903 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:58:53.903 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:58:53.903 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:58:53.903 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:58:53.903 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:58:53.903 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:58:53.903 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:58:53.903 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:58:53.903 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:58:53.903 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:58:53.903 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:58:53.903 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:58:53.904 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:58:53.904 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:58:53.904 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T08:58:53.904 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:58:53.904 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: main() 2024-01-13T08:58:53.904 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:58:53.904 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: r = ctx.func(ctx) 2024-01-13T08:58:53.904 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:58:53.904 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:58:53.904 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:58:53.904 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:58:53.905 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:58:53.905 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:58:53.905 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:58:53.905 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:58:53.905 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:58:53.905 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:58:53.905 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:58:53.905 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:58:53.906 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpesudkjaq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbey7y289:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:58:53.906 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T08:58:52.616263+0000 mgr.smithi077.amvfyi (mgr.14180) 37 : cephadm [INF] Reconfiguring mon.smithi077 (unknown last config time)... 2024-01-13T08:58:53.906 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:58:52.616279+0000 mgr.smithi077.amvfyi (mgr.14180) 38 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:58:53.906 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:52.616826+0000 mon.smithi077 (mon.0) 229 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-13T08:58:53.906 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:52.617488+0000 mon.smithi077 (mon.0) 230 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-13T08:58:53.906 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:52.618058+0000 mon.smithi077 (mon.0) 231 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:58:53.906 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T08:58:52.618609+0000 mgr.smithi077.amvfyi (mgr.14180) 39 : cephadm [INF] Reconfiguring daemon mon.smithi077 on smithi077 2024-01-13T08:58:53.906 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:58:53.086159+0000 mon.smithi077 (mon.0) 232 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T08:58:54.115 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":9,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136331,"num_remapped_pgs":0} 2024-01-13T08:58:54.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:54 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:53.537752+0000 mon.smithi077 (mon.0) 233 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:54.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:54 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:53.726895+0000 mon.smithi077 (mon.0) 234 : audit [DBG] from='client.? 172.21.15.77:0/4042165794' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:58:54.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:54 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:53.822976+0000 mon.smithi077 (mon.0) 235 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:54.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:54 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:58:53.823543+0000 mgr.smithi077.amvfyi (mgr.14180) 40 : cephadm [INF] Reconfiguring mgr.smithi077.amvfyi (unknown last config time)... 2024-01-13T08:58:54.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:54 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:53.823810+0000 mon.smithi077 (mon.0) 236 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi077.amvfyi", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-13T08:58:54.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:54 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:53.824428+0000 mon.smithi077 (mon.0) 237 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-13T08:58:54.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:54 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:53.824913+0000 mon.smithi077 (mon.0) 238 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:58:54.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:54 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:58:53.825360+0000 mgr.smithi077.amvfyi (mgr.14180) 41 : cephadm [INF] Reconfiguring daemon mgr.smithi077.amvfyi on smithi077 2024-01-13T08:58:54.879 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:54 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:53.537752+0000 mon.smithi077 (mon.0) 233 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:54.879 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:54 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:53.726895+0000 mon.smithi077 (mon.0) 234 : audit [DBG] from='client.? 172.21.15.77:0/4042165794' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:58:54.879 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:54 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:53.822976+0000 mon.smithi077 (mon.0) 235 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:54.879 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:54 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T08:58:53.823543+0000 mgr.smithi077.amvfyi (mgr.14180) 40 : cephadm [INF] Reconfiguring mgr.smithi077.amvfyi (unknown last config time)... 2024-01-13T08:58:54.879 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:54 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:53.823810+0000 mon.smithi077 (mon.0) 236 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi077.amvfyi", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-13T08:58:54.880 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:54 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:53.824428+0000 mon.smithi077 (mon.0) 237 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-13T08:58:54.880 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:54 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:53.824913+0000 mon.smithi077 (mon.0) 238 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:58:54.880 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:54 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T08:58:53.825360+0000 mgr.smithi077.amvfyi (mgr.14180) 41 : cephadm [INF] Reconfiguring daemon mgr.smithi077.amvfyi on smithi077 2024-01-13T08:58:55.116 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:58:56.006 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:55 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:54.616668+0000 mgr.smithi077.amvfyi (mgr.14180) 42 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:58:56.007 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:55 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:54.759882+0000 mon.smithi077 (mon.0) 239 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:56.007 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:55 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:58:54.760528+0000 mgr.smithi077.amvfyi (mgr.14180) 43 : cephadm [INF] Reconfiguring alertmanager.smithi077 (dependencies changed)... 2024-01-13T08:58:56.007 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:55 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:58:54.766672+0000 mgr.smithi077.amvfyi (mgr.14180) 44 : cephadm [INF] Reconfiguring daemon alertmanager.smithi077 on smithi077 2024-01-13T08:58:56.129 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:55 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:58:54.616668+0000 mgr.smithi077.amvfyi (mgr.14180) 42 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:58:56.129 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:55 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:54.759882+0000 mon.smithi077 (mon.0) 239 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:56.129 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:55 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T08:58:54.760528+0000 mgr.smithi077.amvfyi (mgr.14180) 43 : cephadm [INF] Reconfiguring alertmanager.smithi077 (dependencies changed)... 2024-01-13T08:58:56.129 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:55 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T08:58:54.766672+0000 mgr.smithi077.amvfyi (mgr.14180) 44 : cephadm [INF] Reconfiguring daemon alertmanager.smithi077 on smithi077 2024-01-13T08:58:57.185 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T08:58:57.656 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":9,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136331,"num_remapped_pgs":0} 2024-01-13T08:58:58.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:57 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:56.617125+0000 mgr.smithi077.amvfyi (mgr.14180) 45 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:58:58.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:57 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:57.191958+0000 mon.smithi077 (mon.0) 240 : audit [DBG] from='client.? 172.21.15.77:0/3252876857' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:58:58.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:57 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:58:56.617125+0000 mgr.smithi077.amvfyi (mgr.14180) 45 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:58:58.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:57 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:57.191958+0000 mon.smithi077 (mon.0) 240 : audit [DBG] from='client.? 172.21.15.77:0/3252876857' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:58:58.657 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:58:59.515 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:59 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:58.182940+0000 mon.smithi077 (mon.0) 241 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:59.515 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:59 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:58:58.183476+0000 mgr.smithi077.amvfyi (mgr.14180) 46 : cephadm [INF] Reconfiguring crash.smithi077 (monmap changed)... 2024-01-13T08:58:59.515 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:59 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:58.183703+0000 mon.smithi077 (mon.0) 242 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi077", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-13T08:58:59.515 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:59 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:58.184287+0000 mon.smithi077 (mon.0) 243 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:58:59.515 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:59 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:58:58.184724+0000 mgr.smithi077.amvfyi (mgr.14180) 47 : cephadm [INF] Reconfiguring daemon crash.smithi077 on smithi077 2024-01-13T08:58:59.515 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:59 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:58:58.617668+0000 mgr.smithi077.amvfyi (mgr.14180) 48 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:58:59.515 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:59 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:58:59.065971+0000 mon.smithi077 (mon.0) 244 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:59.516 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:59 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:58:59.066540+0000 mgr.smithi077.amvfyi (mgr.14180) 49 : cephadm [INF] Reconfiguring grafana.smithi077 (dependencies changed)... 2024-01-13T08:58:59.516 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:58:59 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:58:59.096993+0000 mgr.smithi077.amvfyi (mgr.14180) 50 : cephadm [INF] Reconfiguring daemon grafana.smithi077 on smithi077 2024-01-13T08:58:59.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:59 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:58.182940+0000 mon.smithi077 (mon.0) 241 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:59.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:59 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T08:58:58.183476+0000 mgr.smithi077.amvfyi (mgr.14180) 46 : cephadm [INF] Reconfiguring crash.smithi077 (monmap changed)... 2024-01-13T08:58:59.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:59 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:58.183703+0000 mon.smithi077 (mon.0) 242 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi077", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-13T08:58:59.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:59 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:58.184287+0000 mon.smithi077 (mon.0) 243 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:58:59.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:59 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T08:58:58.184724+0000 mgr.smithi077.amvfyi (mgr.14180) 47 : cephadm [INF] Reconfiguring daemon crash.smithi077 on smithi077 2024-01-13T08:58:59.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:59 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:58:58.617668+0000 mgr.smithi077.amvfyi (mgr.14180) 48 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:58:59.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:59 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:58:59.065971+0000 mon.smithi077 (mon.0) 244 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:58:59.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:59 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T08:58:59.066540+0000 mgr.smithi077.amvfyi (mgr.14180) 49 : cephadm [INF] Reconfiguring grafana.smithi077 (dependencies changed)... 2024-01-13T08:58:59.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:58:59 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T08:58:59.096993+0000 mgr.smithi077.amvfyi (mgr.14180) 50 : cephadm [INF] Reconfiguring daemon grafana.smithi077 on smithi077 2024-01-13T08:59:00.319 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T08:59:00.380 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:00.327100+0000 mon.smithi077 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.77:0/3768809645' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:00.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:00.327100+0000 mon.smithi077 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.77:0/3768809645' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:00.754 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":9,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136331,"num_remapped_pgs":0} 2024-01-13T08:59:01.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:01 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:00.618037+0000 mgr.smithi077.amvfyi (mgr.14180) 51 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:01.755 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:59:01.783 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:01 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:00.618037+0000 mgr.smithi077.amvfyi (mgr.14180) 51 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:03.199 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T08:59:03.544 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:03 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:02.301066+0000 mon.smithi077 (mon.0) 246 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:59:03.544 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:03 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:59:02.301820+0000 mgr.smithi077.amvfyi (mgr.14180) 52 : cephadm [INF] Reconfiguring prometheus.smithi077 (dependencies changed)... 2024-01-13T08:59:03.544 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:03 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:59:02.305975+0000 mgr.smithi077.amvfyi (mgr.14180) 53 : cephadm [INF] Reconfiguring daemon prometheus.smithi077 on smithi077 2024-01-13T08:59:03.544 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:03 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:02.618465+0000 mgr.smithi077.amvfyi (mgr.14180) 54 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:03.544 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:03 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:03.205647+0000 mon.smithi077 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.77:0/2506725266' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:03.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:03 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:02.301066+0000 mon.smithi077 (mon.0) 246 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:59:03.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:03 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T08:59:02.301820+0000 mgr.smithi077.amvfyi (mgr.14180) 52 : cephadm [INF] Reconfiguring prometheus.smithi077 (dependencies changed)... 2024-01-13T08:59:03.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:03 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T08:59:02.305975+0000 mgr.smithi077.amvfyi (mgr.14180) 53 : cephadm [INF] Reconfiguring daemon prometheus.smithi077 on smithi077 2024-01-13T08:59:03.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:03 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:02.618465+0000 mgr.smithi077.amvfyi (mgr.14180) 54 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:03.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:03 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:03.205647+0000 mon.smithi077 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.77:0/2506725266' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:03.672 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":9,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136331,"num_remapped_pgs":0} 2024-01-13T08:59:04.673 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:59:05.881 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:05 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:04.618881+0000 mgr.smithi077.amvfyi (mgr.14180) 55 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:06.129 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:05 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:04.618881+0000 mgr.smithi077.amvfyi (mgr.14180) 55 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:06.308 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T08:59:06.688 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":9,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136331,"num_remapped_pgs":0} 2024-01-13T08:59:07.182 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:06 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:05.917169+0000 mon.smithi077 (mon.0) 248 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:59:07.182 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:06 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T08:59:05.917799+0000 mgr.smithi077.amvfyi (mgr.14180) 56 : cephadm [INF] Reconfiguring crash.smithi141 (monmap changed)... 2024-01-13T08:59:07.182 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:06 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:05.918092+0000 mon.smithi077 (mon.0) 249 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi141", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-13T08:59:07.182 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:06 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:05.918880+0000 mon.smithi077 (mon.0) 250 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:59:07.182 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:06 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T08:59:05.919380+0000 mgr.smithi077.amvfyi (mgr.14180) 57 : cephadm [INF] Reconfiguring daemon crash.smithi141 on smithi141 2024-01-13T08:59:07.182 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:06 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:06.316264+0000 mon.smithi077 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.77:0/2396063378' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:07.182 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:06 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:06.739663+0000 mon.smithi077 (mon.0) 252 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:59:07.182 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:06 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:06.740427+0000 mon.smithi077 (mon.0) 253 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi141.ujrxla", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-13T08:59:07.183 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:06 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:06.741057+0000 mon.smithi077 (mon.0) 254 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-13T08:59:07.183 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:06 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:06.741540+0000 mon.smithi077 (mon.0) 255 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:59:07.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:06 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:05.917169+0000 mon.smithi077 (mon.0) 248 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:59:07.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:06 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:59:05.917799+0000 mgr.smithi077.amvfyi (mgr.14180) 56 : cephadm [INF] Reconfiguring crash.smithi141 (monmap changed)... 2024-01-13T08:59:07.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:06 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:05.918092+0000 mon.smithi077 (mon.0) 249 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi141", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-13T08:59:07.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:06 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:05.918880+0000 mon.smithi077 (mon.0) 250 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:59:07.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:06 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:59:05.919380+0000 mgr.smithi077.amvfyi (mgr.14180) 57 : cephadm [INF] Reconfiguring daemon crash.smithi141 on smithi141 2024-01-13T08:59:07.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:06 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:06.316264+0000 mon.smithi077 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.77:0/2396063378' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:07.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:06 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:06.739663+0000 mon.smithi077 (mon.0) 252 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:59:07.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:06 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:06.740427+0000 mon.smithi077 (mon.0) 253 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi141.ujrxla", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-13T08:59:07.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:06 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:06.741057+0000 mon.smithi077 (mon.0) 254 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-13T08:59:07.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:06 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:06.741540+0000 mon.smithi077 (mon.0) 255 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:59:07.690 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:59:08.014 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:07 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:06.619402+0000 mgr.smithi077.amvfyi (mgr.14180) 58 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:08.014 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:07 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:59:06.740146+0000 mgr.smithi077.amvfyi (mgr.14180) 59 : cephadm [INF] Reconfiguring mgr.smithi141.ujrxla (monmap changed)... 2024-01-13T08:59:08.014 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:07 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:59:06.741985+0000 mgr.smithi077.amvfyi (mgr.14180) 60 : cephadm [INF] Reconfiguring daemon mgr.smithi141.ujrxla on smithi141 2024-01-13T08:59:08.015 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:07 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:07.588980+0000 mon.smithi077 (mon.0) 256 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:59:08.015 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:07 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:07.589940+0000 mon.smithi077 (mon.0) 257 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-13T08:59:08.015 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:07 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:07.590650+0000 mon.smithi077 (mon.0) 258 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-13T08:59:08.015 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:07 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:07.591288+0000 mon.smithi077 (mon.0) 259 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:59:08.129 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:07 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:06.619402+0000 mgr.smithi077.amvfyi (mgr.14180) 58 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:08.129 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:07 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T08:59:06.740146+0000 mgr.smithi077.amvfyi (mgr.14180) 59 : cephadm [INF] Reconfiguring mgr.smithi141.ujrxla (monmap changed)... 2024-01-13T08:59:08.129 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:07 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T08:59:06.741985+0000 mgr.smithi077.amvfyi (mgr.14180) 60 : cephadm [INF] Reconfiguring daemon mgr.smithi141.ujrxla on smithi141 2024-01-13T08:59:08.129 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:07 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:07.588980+0000 mon.smithi077 (mon.0) 256 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:59:08.129 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:07 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:07.589940+0000 mon.smithi077 (mon.0) 257 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-13T08:59:08.129 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:07 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:07.590650+0000 mon.smithi077 (mon.0) 258 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-13T08:59:08.129 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:07 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:07.591288+0000 mon.smithi077 (mon.0) 259 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:59:09.009 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T08:59:09.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:08 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:59:07.589593+0000 mgr.smithi077.amvfyi (mgr.14180) 61 : cephadm [INF] Reconfiguring mon.smithi141 (monmap changed)... 2024-01-13T08:59:09.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:08 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:59:07.591871+0000 mgr.smithi077.amvfyi (mgr.14180) 62 : cephadm [INF] Reconfiguring daemon mon.smithi141 on smithi141 2024-01-13T08:59:09.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:08 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:08.457008+0000 mon.smithi077 (mon.0) 260 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:59:09.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:08 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:08.459472+0000 mon.smithi077 (mon.0) 261 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-13T08:59:09.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:08 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:08.460661+0000 mon.smithi077 (mon.0) 262 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi077:9093"}]: dispatch 2024-01-13T08:59:09.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:08 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:08.464291+0000 mon.smithi077 (mon.0) 263 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:59:09.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:08 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:08.469276+0000 mon.smithi077 (mon.0) 264 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-13T08:59:09.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:08 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:08.471967+0000 mon.smithi077 (mon.0) 265 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi077:3000"}]: dispatch 2024-01-13T08:59:09.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:08 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:08.476338+0000 mon.smithi077 (mon.0) 266 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:59:09.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:08 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:08.480242+0000 mon.smithi077 (mon.0) 267 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-01-13T08:59:09.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:08 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:08.482498+0000 mon.smithi077 (mon.0) 268 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi077:9095"}]: dispatch 2024-01-13T08:59:09.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:08 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:08.487124+0000 mon.smithi077 (mon.0) 269 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:59:09.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:08 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:08.489796+0000 mon.smithi077 (mon.0) 270 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:59:09.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:08 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T08:59:07.589593+0000 mgr.smithi077.amvfyi (mgr.14180) 61 : cephadm [INF] Reconfiguring mon.smithi141 (monmap changed)... 2024-01-13T08:59:09.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:08 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T08:59:07.591871+0000 mgr.smithi077.amvfyi (mgr.14180) 62 : cephadm [INF] Reconfiguring daemon mon.smithi141 on smithi141 2024-01-13T08:59:09.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:08 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:08.457008+0000 mon.smithi077 (mon.0) 260 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:59:09.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:08 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:08.459472+0000 mon.smithi077 (mon.0) 261 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-13T08:59:09.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:08 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:08.460661+0000 mon.smithi077 (mon.0) 262 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi077:9093"}]: dispatch 2024-01-13T08:59:09.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:08 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:08.464291+0000 mon.smithi077 (mon.0) 263 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:59:09.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:08 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:08.469276+0000 mon.smithi077 (mon.0) 264 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-13T08:59:09.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:08 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:08.471967+0000 mon.smithi077 (mon.0) 265 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi077:3000"}]: dispatch 2024-01-13T08:59:09.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:08 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:08.476338+0000 mon.smithi077 (mon.0) 266 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:59:09.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:08 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:08.480242+0000 mon.smithi077 (mon.0) 267 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-01-13T08:59:09.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:08 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:08.482498+0000 mon.smithi077 (mon.0) 268 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi077:9095"}]: dispatch 2024-01-13T08:59:09.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:08 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:08.487124+0000 mon.smithi077 (mon.0) 269 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:59:09.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:08 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:08.489796+0000 mon.smithi077 (mon.0) 270 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:59:09.567 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":9,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136331,"num_remapped_pgs":0} 2024-01-13T08:59:10.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:09 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:08.459847+0000 mgr.smithi077.amvfyi (mgr.14180) 63 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-13T08:59:10.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:09 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:08.460962+0000 mgr.smithi077.amvfyi (mgr.14180) 64 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi077:9093"}]: dispatch 2024-01-13T08:59:10.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:09 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:08.469597+0000 mgr.smithi077.amvfyi (mgr.14180) 65 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-13T08:59:10.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:09 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:08.472322+0000 mgr.smithi077.amvfyi (mgr.14180) 66 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi077:3000"}]: dispatch 2024-01-13T08:59:10.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:09 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:08.480593+0000 mgr.smithi077.amvfyi (mgr.14180) 67 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-01-13T08:59:10.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:09 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:08.482843+0000 mgr.smithi077.amvfyi (mgr.14180) 68 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi077:9095"}]: dispatch 2024-01-13T08:59:10.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:09 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:08.620058+0000 mgr.smithi077.amvfyi (mgr.14180) 69 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:10.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:09 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:09.017078+0000 mon.smithi077 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.77:0/627249111' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:10.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:09 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:09.564129+0000 mon.smithi077 (mon.0) 272 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:59:10.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:09 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:08.459847+0000 mgr.smithi077.amvfyi (mgr.14180) 63 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-13T08:59:10.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:09 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:08.460962+0000 mgr.smithi077.amvfyi (mgr.14180) 64 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi077:9093"}]: dispatch 2024-01-13T08:59:10.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:09 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:08.469597+0000 mgr.smithi077.amvfyi (mgr.14180) 65 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-13T08:59:10.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:09 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:08.472322+0000 mgr.smithi077.amvfyi (mgr.14180) 66 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi077:3000"}]: dispatch 2024-01-13T08:59:10.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:09 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:08.480593+0000 mgr.smithi077.amvfyi (mgr.14180) 67 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-01-13T08:59:10.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:09 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:08.482843+0000 mgr.smithi077.amvfyi (mgr.14180) 68 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi077:9095"}]: dispatch 2024-01-13T08:59:10.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:09 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:08.620058+0000 mgr.smithi077.amvfyi (mgr.14180) 69 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:10.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:09 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:09.017078+0000 mon.smithi077 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.77:0/627249111' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:10.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:09 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:09.564129+0000 mon.smithi077 (mon.0) 272 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:59:10.569 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:59:12.279 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:11 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:10.620459+0000 mgr.smithi077.amvfyi (mgr.14180) 70 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:12.279 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:11 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:11.274179+0000 mon.smithi077 (mon.0) 273 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:59:12.279 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:11 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:11.278472+0000 mon.smithi077 (mon.0) 274 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:59:12.279 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:11 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:11.279201+0000 mon.smithi077 (mon.0) 275 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:59:12.279 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:11 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:11.279833+0000 mon.smithi077 (mon.0) 276 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T08:59:12.279 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:11 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:11.285564+0000 mon.smithi077 (mon.0) 277 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:59:12.280 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:11 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:11.289092+0000 mon.smithi077 (mon.0) 278 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T08:59:12.280 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:11 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:11.291759+0000 mon.smithi077 (mon.0) 279 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:59:12.280 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:11 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:11.292681+0000 mon.smithi077 (mon.0) 280 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:59:12.280 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:11 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:11.292860+0000 mon.smithi077 (mon.0) 281 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:59:12.280 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:11 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:11.293279+0000 mon.smithi077 (mon.0) 282 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:59:12.290 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:11 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:10.620459+0000 mgr.smithi077.amvfyi (mgr.14180) 70 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:12.290 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:11 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:11.274179+0000 mon.smithi077 (mon.0) 273 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:59:12.290 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:11 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:11.278472+0000 mon.smithi077 (mon.0) 274 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:59:12.290 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:11 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:11.279201+0000 mon.smithi077 (mon.0) 275 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:59:12.290 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:11 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:11.279833+0000 mon.smithi077 (mon.0) 276 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T08:59:12.290 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:11 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:11.285564+0000 mon.smithi077 (mon.0) 277 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:59:12.290 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:11 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:11.289092+0000 mon.smithi077 (mon.0) 278 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T08:59:12.290 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:11 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:11.291759+0000 mon.smithi077 (mon.0) 279 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:59:12.291 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:11 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:11.292681+0000 mon.smithi077 (mon.0) 280 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:59:12.291 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:11 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:11.292860+0000 mon.smithi077 (mon.0) 281 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:59:12.291 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:11 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:11.293279+0000 mon.smithi077 (mon.0) 282 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:59:12.480 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T08:59:12.888 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":9,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136331,"num_remapped_pgs":0} 2024-01-13T08:59:13.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:12 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:11.285835+0000 mgr.smithi077.amvfyi (mgr.14180) 71 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:13.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:12 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:12.284410+0000 mon.smithi077 (mon.0) 283 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T08:59:13.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:12 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:12.284437+0000 mon.smithi077 (mon.0) 284 : cluster [INF] Cluster is now healthy 2024-01-13T08:59:13.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:12 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:12.487754+0000 mon.smithi077 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.77:0/1681949427' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:13.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:12 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:11.285835+0000 mgr.smithi077.amvfyi (mgr.14180) 71 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:13.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:12 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:12.284410+0000 mon.smithi077 (mon.0) 283 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T08:59:13.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:12 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:12.284437+0000 mon.smithi077 (mon.0) 284 : cluster [INF] Cluster is now healthy 2024-01-13T08:59:13.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:12 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:12.487754+0000 mon.smithi077 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.77:0/1681949427' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:13.889 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:59:14.129 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:13 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:13.327717+0000 mon.smithi141 (mon.1) 4 : audit [INF] from='client.? 172.21.15.141:0/392021653' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae8ce16a-513d-46ad-b1f2-d22780aa44cb"}]: dispatch 2024-01-13T08:59:14.130 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:13 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:13.328104+0000 mon.smithi077 (mon.0) 286 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae8ce16a-513d-46ad-b1f2-d22780aa44cb"}]: dispatch 2024-01-13T08:59:14.130 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:13 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:13.336016+0000 mon.smithi077 (mon.0) 287 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae8ce16a-513d-46ad-b1f2-d22780aa44cb"}]': finished 2024-01-13T08:59:14.130 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:13 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:13.336065+0000 mon.smithi077 (mon.0) 288 : cluster [DBG] osdmap e10: 1 total, 0 up, 1 in 2024-01-13T08:59:14.130 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:13 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:13.336157+0000 mon.smithi077 (mon.0) 289 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:59:14.130 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:13 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:13.614373+0000 mon.smithi077 (mon.0) 290 : audit [INF] from='client.? 172.21.15.77:0/1420704769' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec0b7742-b5ca-4974-9912-71048c0afbb3"}]: dispatch 2024-01-13T08:59:14.130 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:13 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:13.620619+0000 mon.smithi077 (mon.0) 291 : audit [INF] from='client.? 172.21.15.77:0/1420704769' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec0b7742-b5ca-4974-9912-71048c0afbb3"}]': finished 2024-01-13T08:59:14.130 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:13 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:13.620692+0000 mon.smithi077 (mon.0) 292 : cluster [DBG] osdmap e11: 2 total, 0 up, 2 in 2024-01-13T08:59:14.130 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:13 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:13.620800+0000 mon.smithi077 (mon.0) 293 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:59:14.130 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:13 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:13.621147+0000 mon.smithi077 (mon.0) 294 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-13T08:59:14.130 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:13 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:13.623884+0000 mon.smithi141 (mon.1) 5 : audit [INF] from='client.? 172.21.15.141:0/3555517002' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:59:14.130 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:13 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:13.624185+0000 mon.smithi077 (mon.0) 295 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:59:14.130 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:13 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:13.629804+0000 mon.smithi077 (mon.0) 296 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:59:14.130 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:13 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:13.629851+0000 mon.smithi077 (mon.0) 297 : cluster [DBG] osdmap e12: 1 total, 0 up, 1 in 2024-01-13T08:59:14.131 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:13 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:13.629977+0000 mon.smithi077 (mon.0) 298 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-13T08:59:14.131 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:13 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:13.909469+0000 mon.smithi077 (mon.0) 299 : audit [INF] from='client.? 172.21.15.77:0/3120135921' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:59:14.131 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:13 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:13.913807+0000 mon.smithi077 (mon.0) 300 : audit [INF] from='client.? 172.21.15.77:0/3120135921' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:59:14.131 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:13 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:13.913867+0000 mon.smithi077 (mon.0) 301 : cluster [DBG] osdmap e13: 0 total, 0 up, 0 in 2024-01-13T08:59:14.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:13 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:13.327717+0000 mon.smithi141 (mon.1) 4 : audit [INF] from='client.? 172.21.15.141:0/392021653' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae8ce16a-513d-46ad-b1f2-d22780aa44cb"}]: dispatch 2024-01-13T08:59:14.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:13 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:13.328104+0000 mon.smithi077 (mon.0) 286 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae8ce16a-513d-46ad-b1f2-d22780aa44cb"}]: dispatch 2024-01-13T08:59:14.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:13 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:13.336016+0000 mon.smithi077 (mon.0) 287 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae8ce16a-513d-46ad-b1f2-d22780aa44cb"}]': finished 2024-01-13T08:59:14.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:13 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:13.336065+0000 mon.smithi077 (mon.0) 288 : cluster [DBG] osdmap e10: 1 total, 0 up, 1 in 2024-01-13T08:59:14.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:13 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:13.336157+0000 mon.smithi077 (mon.0) 289 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:59:14.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:13 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:13.614373+0000 mon.smithi077 (mon.0) 290 : audit [INF] from='client.? 172.21.15.77:0/1420704769' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec0b7742-b5ca-4974-9912-71048c0afbb3"}]: dispatch 2024-01-13T08:59:14.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:13 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:13.620619+0000 mon.smithi077 (mon.0) 291 : audit [INF] from='client.? 172.21.15.77:0/1420704769' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec0b7742-b5ca-4974-9912-71048c0afbb3"}]': finished 2024-01-13T08:59:14.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:13 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:13.620692+0000 mon.smithi077 (mon.0) 292 : cluster [DBG] osdmap e11: 2 total, 0 up, 2 in 2024-01-13T08:59:14.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:13 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:13.620800+0000 mon.smithi077 (mon.0) 293 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:59:14.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:13 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:13.621147+0000 mon.smithi077 (mon.0) 294 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-13T08:59:14.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:13 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:13.623884+0000 mon.smithi141 (mon.1) 5 : audit [INF] from='client.? 172.21.15.141:0/3555517002' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:59:14.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:13 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:13.624185+0000 mon.smithi077 (mon.0) 295 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:59:14.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:13 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:13.629804+0000 mon.smithi077 (mon.0) 296 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:59:14.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:13 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:13.629851+0000 mon.smithi077 (mon.0) 297 : cluster [DBG] osdmap e12: 1 total, 0 up, 1 in 2024-01-13T08:59:14.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:13 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:13.629977+0000 mon.smithi077 (mon.0) 298 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-13T08:59:14.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:13 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:13.909469+0000 mon.smithi077 (mon.0) 299 : audit [INF] from='client.? 172.21.15.77:0/3120135921' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:59:14.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:13 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:13.913807+0000 mon.smithi077 (mon.0) 300 : audit [INF] from='client.? 172.21.15.77:0/3120135921' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:59:14.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:13 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:13.913867+0000 mon.smithi077 (mon.0) 301 : cluster [DBG] osdmap e13: 0 total, 0 up, 0 in 2024-01-13T08:59:15.323 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:13.286090+0000 mgr.smithi077.amvfyi (mgr.14180) 72 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:15.323 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:59:14.023609+0000 mgr.smithi077.amvfyi (mgr.14180) 73 : cephadm [ERR] executing create_from_spec_one(([('smithi077', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi141', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:59:15.323 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T08:59:15.323 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:59:15.323 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return f(*arg) 2024-01-13T08:59:15.323 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:59:15.323 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:59:15.323 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:59:15.323 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: code, '\n'.join(err))) 2024-01-13T08:59:15.323 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T08:59:15.324 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpouclylzf:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmplqo2admp:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:59:15.324 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:59:15.324 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:59:15.324 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:59:15.324 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new ae8ce16a-513d-46ad-b1f2-d22780aa44cb 2024-01-13T08:59:15.324 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:59:15.324 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:59:15.324 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:59:15.324 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:59:15.324 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:59:15.324 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.prepare() 2024-01-13T08:59:15.324 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:15.324 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:15.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:59:15.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:59:15.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:59:15.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:59:15.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:59:15.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:59:15.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:59:15.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T08:59:15.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:59:15.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T08:59:15.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:59:15.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:59:15.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:59:15.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:59:15.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:59:15.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:59:15.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:59:15.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:59:15.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:59:15.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:59:15.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T08:59:15.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:59:15.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:59:15.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:59:15.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T08:59:15.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:15.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:15.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:59:15.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:59:15.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:59:15.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:59:15.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:15.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:15.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:59:15.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:59:15.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:59:15.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:59:15.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:59:15.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:59:15.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:59:15.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:59:15.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:15.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:15.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:59:15.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:59:15.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:59:15.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:59:15.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:59:15.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T08:59:15.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:59:15.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: main() 2024-01-13T08:59:15.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:59:15.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: r = ctx.func(ctx) 2024-01-13T08:59:15.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:59:15.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:59:15.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:59:15.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:59:15.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:59:15.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:59:15.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:59:15.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:59:15.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:59:15.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:59:15.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:59:15.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:59:15.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpouclylzf:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmplqo2admp:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:59:15.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:14.588063+0000 mon.smithi077 (mon.0) 302 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:59:15.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:13.286090+0000 mgr.smithi077.amvfyi (mgr.14180) 72 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:15.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T08:59:14.023609+0000 mgr.smithi077.amvfyi (mgr.14180) 73 : cephadm [ERR] executing create_from_spec_one(([('smithi077', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi141', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:59:15.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T08:59:15.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:59:15.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return f(*arg) 2024-01-13T08:59:15.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:59:15.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:59:15.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:59:15.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: code, '\n'.join(err))) 2024-01-13T08:59:15.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T08:59:15.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpouclylzf:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmplqo2admp:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:59:15.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:59:15.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:59:15.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:59:15.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new ae8ce16a-513d-46ad-b1f2-d22780aa44cb 2024-01-13T08:59:15.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:59:15.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:59:15.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:59:15.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:59:15.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:59:15.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.prepare() 2024-01-13T08:59:15.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:15.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:15.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:59:15.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:59:15.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:59:15.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:59:15.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:59:15.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:59:15.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:59:15.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T08:59:15.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:59:15.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T08:59:15.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:59:15.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:59:15.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:59:15.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:59:15.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:59:15.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:59:15.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:59:15.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:59:15.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:59:15.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:59:15.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T08:59:15.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:59:15.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:59:15.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:59:15.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T08:59:15.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:15.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:15.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:59:15.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:59:15.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:59:15.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:59:15.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:15.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:15.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:59:15.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:59:15.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:59:15.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:59:15.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:59:15.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:59:15.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:59:15.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:59:15.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:15.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:15.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:59:15.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:59:15.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:59:15.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:59:15.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:59:15.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T08:59:15.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:59:15.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: main() 2024-01-13T08:59:15.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:59:15.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: r = ctx.func(ctx) 2024-01-13T08:59:15.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:59:15.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:59:15.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:59:15.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:59:15.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:59:15.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:59:15.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:59:15.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:59:15.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:59:15.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:59:15.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:59:15.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:59:15.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpouclylzf:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmplqo2admp:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:59:15.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:14.588063+0000 mon.smithi077 (mon.0) 302 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:59:15.731 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T08:59:16.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:59:14.584461+0000 mgr.smithi077.amvfyi (mgr.14180) 74 : cephadm [ERR] executing create_from_spec_one(([('smithi077', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi141', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:59:16.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T08:59:16.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:59:16.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return f(*arg) 2024-01-13T08:59:16.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:59:16.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:59:16.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:59:16.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: code, '\n'.join(err))) 2024-01-13T08:59:16.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi077/config 2024-01-13T08:59:16.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp7x2n6st3:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpcbou5ea9:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:59:16.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:59:16.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:59:16.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:59:16.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new ec0b7742-b5ca-4974-9912-71048c0afbb3 2024-01-13T08:59:16.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:59:16.079 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 --yes-i-really-mean-it 2024-01-13T08:59:16.079 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr stderr: purged osd.1 2024-01-13T08:59:16.079 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:59:16.079 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:59:16.079 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.prepare() 2024-01-13T08:59:16.079 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:16.079 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:16.079 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:59:16.079 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:59:16.079 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:59:16.079 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:59:16.079 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:59:16.079 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:59:16.079 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:59:16.080 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T08:59:16.080 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:59:16.080 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T08:59:16.080 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:59:16.080 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:59:16.080 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:59:16.080 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:59:16.080 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:59:16.080 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:59:16.080 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:59:16.080 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:59:16.080 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:59:16.080 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:59:16.081 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T08:59:16.081 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:59:16.081 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:59:16.081 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:59:16.081 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T08:59:16.081 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:16.081 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:16.081 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:59:16.081 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:59:16.081 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:59:16.081 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:59:16.081 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:16.081 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:16.081 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:59:16.082 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:59:16.082 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:59:16.082 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:59:16.082 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:59:16.082 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:59:16.082 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:59:16.082 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:59:16.082 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:16.082 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:16.082 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:59:16.082 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:59:16.082 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:59:16.082 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:59:16.082 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 1 2024-01-13T08:59:16.083 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T08:59:16.083 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:59:16.083 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: main() 2024-01-13T08:59:16.083 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:59:16.083 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: r = ctx.func(ctx) 2024-01-13T08:59:16.083 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:59:16.083 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:59:16.083 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:59:16.083 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:59:16.083 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:59:16.083 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:59:16.083 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:59:16.083 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:59:16.083 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:59:16.083 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:59:16.084 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:59:16.084 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:59:16.084 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp7x2n6st3:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpcbou5ea9:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:59:16.084 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:59:14.586266+0000 mgr.smithi077.amvfyi (mgr.14180) 75 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T08:59:16.084 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: service_id: all-available-devices 2024-01-13T08:59:16.084 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: service_name: osd.all-available-devices 2024-01-13T08:59:16.084 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: placement: 2024-01-13T08:59:16.084 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: host_pattern: '*' 2024-01-13T08:59:16.084 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: spec: 2024-01-13T08:59:16.084 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: data_devices: 2024-01-13T08:59:16.084 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: all: true 2024-01-13T08:59:16.084 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: filter_logic: AND 2024-01-13T08:59:16.084 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: objectstore: bluestore 2024-01-13T08:59:16.085 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T08:59:16.085 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpouclylzf:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmplqo2admp:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:59:16.085 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:59:16.085 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:59:16.085 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:59:16.085 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new ae8ce16a-513d-46ad-b1f2-d22780aa44cb 2024-01-13T08:59:16.085 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:59:16.085 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:59:16.085 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:59:16.085 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:59:16.085 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:59:16.085 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.prepare() 2024-01-13T08:59:16.085 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:16.085 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:16.085 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:59:16.086 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:59:16.086 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:59:16.086 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:59:16.086 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:59:16.086 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:59:16.086 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:59:16.086 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T08:59:16.086 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:59:16.086 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T08:59:16.086 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:59:16.086 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:59:16.086 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:59:16.086 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:59:16.086 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:59:16.086 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:59:16.087 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:59:16.087 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:59:16.087 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:59:16.087 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:59:16.087 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T08:59:16.087 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:59:16.087 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:59:16.087 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:59:16.087 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T08:59:16.087 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:16.087 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:16.087 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:59:16.087 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:59:16.087 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:59:16.088 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:59:16.088 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:16.088 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:16.088 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:59:16.088 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:59:16.088 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:59:16.088 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:59:16.088 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:59:16.088 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:59:16.088 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:59:16.088 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:59:16.088 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:16.088 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:16.089 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:59:16.089 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:59:16.089 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:59:16.089 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:59:16.089 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:59:16.089 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T08:59:16.089 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:59:16.089 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: main() 2024-01-13T08:59:16.089 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:59:16.089 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: r = ctx.func(ctx) 2024-01-13T08:59:16.089 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:59:16.089 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:59:16.089 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:59:16.090 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:59:16.090 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:59:16.090 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:59:16.090 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:59:16.090 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:59:16.090 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:59:16.090 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:59:16.090 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:59:16.090 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:59:16.090 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpouclylzf:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmplqo2admp:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:59:16.090 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T08:59:16.090 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T08:59:16.090 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: if self._apply_service(spec): 2024-01-13T08:59:16.090 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T08:59:16.091 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T08:59:16.091 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T08:59:16.091 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T08:59:16.091 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T08:59:16.091 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T08:59:16.091 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T08:59:16.091 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T08:59:16.091 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T08:59:16.091 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: raise self._value 2024-01-13T08:59:16.091 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T08:59:16.091 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: result = (True, func(*args, **kwds)) 2024-01-13T08:59:16.091 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T08:59:16.091 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return list(map(*args)) 2024-01-13T08:59:16.091 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:59:16.091 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return f(*arg) 2024-01-13T08:59:16.092 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:59:16.092 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:59:16.092 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:59:16.092 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: code, '\n'.join(err))) 2024-01-13T08:59:16.092 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T08:59:16.092 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpouclylzf:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmplqo2admp:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:59:16.092 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:59:16.092 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:59:16.092 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:59:16.092 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new ae8ce16a-513d-46ad-b1f2-d22780aa44cb 2024-01-13T08:59:16.092 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:59:16.092 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:59:16.092 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:59:16.092 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:59:16.093 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:59:16.093 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.prepare() 2024-01-13T08:59:16.093 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:16.093 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:16.093 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:59:16.093 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:59:16.093 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:59:16.093 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:59:16.093 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:59:16.093 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:59:16.093 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:59:16.093 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T08:59:16.093 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:59:16.093 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T08:59:16.093 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:59:16.093 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:59:16.094 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:59:16.094 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:59:16.094 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:59:16.094 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:59:16.094 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:59:16.095 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:59:16.095 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:59:16.095 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:59:16.095 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T08:59:16.095 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:59:16.095 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:59:16.095 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:59:16.095 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T08:59:16.095 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:16.095 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:16.095 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:59:16.095 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:59:16.095 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:59:16.095 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:59:16.096 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:16.096 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:16.096 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:59:16.096 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:59:16.096 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:59:16.096 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:59:16.096 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:59:16.096 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:59:16.096 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:59:16.096 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:59:16.096 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:16.096 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:16.096 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:59:16.096 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:59:16.097 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:59:16.097 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:59:16.097 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:59:16.097 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T08:59:16.097 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:59:16.097 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: main() 2024-01-13T08:59:16.097 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:59:16.097 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: r = ctx.func(ctx) 2024-01-13T08:59:16.097 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:59:16.097 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:59:16.097 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:59:16.097 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:59:16.097 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:59:16.097 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:59:16.098 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:59:16.098 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:59:16.098 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:59:16.098 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:59:16.098 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:59:16.098 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:59:16.098 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpouclylzf:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmplqo2admp:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:59:16.098 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:14.586903+0000 mgr.smithi077.amvfyi (mgr.14180) 76 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:16.098 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:14.951757+0000 mon.smithi077 (mon.0) 303 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T08:59:16.098 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:15.738844+0000 mon.smithi077 (mon.0) 304 : audit [DBG] from='client.? 172.21.15.77:0/97905400' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:16.098 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:15.753991+0000 mon.smithi077 (mon.0) 305 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:59:16.202 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T08:59:16.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T08:59:14.584461+0000 mgr.smithi077.amvfyi (mgr.14180) 74 : cephadm [ERR] executing create_from_spec_one(([('smithi077', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi141', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:59:16.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T08:59:16.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:59:16.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return f(*arg) 2024-01-13T08:59:16.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:59:16.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:59:16.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:59:16.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: code, '\n'.join(err))) 2024-01-13T08:59:16.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi077/config 2024-01-13T08:59:16.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp7x2n6st3:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpcbou5ea9:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:59:16.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:59:16.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:59:16.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:59:16.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new ec0b7742-b5ca-4974-9912-71048c0afbb3 2024-01-13T08:59:16.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:59:16.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 --yes-i-really-mean-it 2024-01-13T08:59:16.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr stderr: purged osd.1 2024-01-13T08:59:16.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:59:16.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:59:16.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.prepare() 2024-01-13T08:59:16.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:16.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:16.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:59:16.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:59:16.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:59:16.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:59:16.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:59:16.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:59:16.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:59:16.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T08:59:16.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:59:16.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T08:59:16.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:59:16.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:59:16.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:59:16.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:59:16.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:59:16.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:59:16.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:59:16.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:59:16.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:59:16.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:59:16.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T08:59:16.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:59:16.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:59:16.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:59:16.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T08:59:16.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:16.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:16.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:59:16.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:59:16.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:59:16.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:59:16.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:16.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:16.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:59:16.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:59:16.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:59:16.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:59:16.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:59:16.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:59:16.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:59:16.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:59:16.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:16.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:16.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:59:16.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:59:16.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:59:16.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:59:16.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 1 2024-01-13T08:59:16.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T08:59:16.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:59:16.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: main() 2024-01-13T08:59:16.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:59:16.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: r = ctx.func(ctx) 2024-01-13T08:59:16.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:59:16.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:59:16.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:59:16.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:59:16.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:59:16.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:59:16.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:59:16.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:59:16.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:59:16.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:59:16.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:59:16.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:59:16.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp7x2n6st3:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpcbou5ea9:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:59:16.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T08:59:14.586266+0000 mgr.smithi077.amvfyi (mgr.14180) 75 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T08:59:16.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: service_id: all-available-devices 2024-01-13T08:59:16.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: service_name: osd.all-available-devices 2024-01-13T08:59:16.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: placement: 2024-01-13T08:59:16.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: host_pattern: '*' 2024-01-13T08:59:16.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: spec: 2024-01-13T08:59:16.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: data_devices: 2024-01-13T08:59:16.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: all: true 2024-01-13T08:59:16.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: filter_logic: AND 2024-01-13T08:59:16.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: objectstore: bluestore 2024-01-13T08:59:16.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T08:59:16.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpouclylzf:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmplqo2admp:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:59:16.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:59:16.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:59:16.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:59:16.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new ae8ce16a-513d-46ad-b1f2-d22780aa44cb 2024-01-13T08:59:16.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:59:16.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:59:16.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:59:16.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:59:16.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:59:16.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.prepare() 2024-01-13T08:59:16.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:16.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:16.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:59:16.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:59:16.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:59:16.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:59:16.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:59:16.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:59:16.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:59:16.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T08:59:16.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:59:16.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T08:59:16.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:59:16.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:59:16.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:59:16.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:59:16.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:59:16.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:59:16.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:59:16.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:59:16.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:59:16.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:59:16.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T08:59:16.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:59:16.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:59:16.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:59:16.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T08:59:16.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:16.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:16.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:59:16.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:59:16.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:59:16.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:59:16.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:16.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:16.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:59:16.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:59:16.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:59:16.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:59:16.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:59:16.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:59:16.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:59:16.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:59:16.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:16.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:16.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:59:16.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:59:16.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:59:16.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:59:16.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:59:16.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T08:59:16.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:59:16.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: main() 2024-01-13T08:59:16.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:59:16.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: r = ctx.func(ctx) 2024-01-13T08:59:16.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:59:16.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:59:16.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:59:16.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:59:16.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:59:16.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:59:16.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:59:16.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:59:16.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:59:16.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:59:16.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:59:16.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:59:16.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpouclylzf:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmplqo2admp:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:59:16.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T08:59:16.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T08:59:16.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: if self._apply_service(spec): 2024-01-13T08:59:16.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T08:59:16.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T08:59:16.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T08:59:16.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T08:59:16.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T08:59:16.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T08:59:16.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T08:59:16.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T08:59:16.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T08:59:16.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: raise self._value 2024-01-13T08:59:16.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T08:59:16.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: result = (True, func(*args, **kwds)) 2024-01-13T08:59:16.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T08:59:16.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return list(map(*args)) 2024-01-13T08:59:16.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:59:16.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return f(*arg) 2024-01-13T08:59:16.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:59:16.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:59:16.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:59:16.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: code, '\n'.join(err))) 2024-01-13T08:59:16.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T08:59:16.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpouclylzf:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmplqo2admp:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:59:16.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:59:16.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:59:16.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:59:16.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new ae8ce16a-513d-46ad-b1f2-d22780aa44cb 2024-01-13T08:59:16.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:59:16.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:59:16.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:59:16.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:59:16.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:59:16.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.prepare() 2024-01-13T08:59:16.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:16.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:16.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:59:16.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:59:16.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:59:16.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:59:16.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:59:16.397 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:59:16.397 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:59:16.397 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T08:59:16.397 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:59:16.397 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T08:59:16.397 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:59:16.397 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:59:16.397 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:59:16.398 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:59:16.398 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:59:16.398 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:59:16.398 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:59:16.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:59:16.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:59:16.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:59:16.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T08:59:16.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:59:16.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:59:16.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:59:16.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T08:59:16.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:16.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:16.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:59:16.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:59:16.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:59:16.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:59:16.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:16.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:16.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:59:16.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:59:16.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:59:16.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:59:16.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:59:16.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:59:16.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:59:16.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:59:16.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:16.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:16.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:59:16.401 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:59:16.401 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:59:16.401 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:59:16.401 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:59:16.401 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T08:59:16.401 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:59:16.401 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: main() 2024-01-13T08:59:16.401 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:59:16.401 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: r = ctx.func(ctx) 2024-01-13T08:59:16.401 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:59:16.401 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:59:16.401 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:59:16.401 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:59:16.401 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:59:16.402 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:59:16.402 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:59:16.402 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:59:16.402 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:59:16.402 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:59:16.402 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:59:16.402 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:59:16.402 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpouclylzf:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmplqo2admp:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:59:16.402 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:14.586903+0000 mgr.smithi077.amvfyi (mgr.14180) 76 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:16.402 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:14.951757+0000 mon.smithi077 (mon.0) 303 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T08:59:16.402 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:15.738844+0000 mon.smithi077 (mon.0) 304 : audit [DBG] from='client.? 172.21.15.77:0/97905400' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:16.402 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:15.753991+0000 mon.smithi077 (mon.0) 305 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:59:17.203 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:59:18.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:17 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:16.587358+0000 mgr.smithi077.amvfyi (mgr.14180) 77 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:18.293 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:17 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:17.865406+0000 mon.smithi077 (mon.0) 306 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:59:18.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:17 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:16.587358+0000 mgr.smithi077.amvfyi (mgr.14180) 77 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:18.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:17 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:17.865406+0000 mon.smithi077 (mon.0) 306 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:59:18.824 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T08:59:19.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:18 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:18.212946+0000 mon.smithi077 (mon.0) 307 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:59:19.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:18 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:18.213594+0000 mon.smithi077 (mon.0) 308 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T08:59:19.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:18 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:18.217399+0000 mon.smithi077 (mon.0) 309 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:59:19.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:18 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:18.220710+0000 mon.smithi077 (mon.0) 310 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T08:59:19.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:18 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:18.223246+0000 mon.smithi077 (mon.0) 311 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:59:19.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:18 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:18.224199+0000 mon.smithi077 (mon.0) 312 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:59:19.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:18 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:18.224406+0000 mon.smithi077 (mon.0) 313 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:59:19.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:18 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:18.224806+0000 mon.smithi077 (mon.0) 314 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:59:19.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:18 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:18.832051+0000 mon.smithi141 (mon.1) 6 : audit [DBG] from='client.? 172.21.15.77:0/516060255' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:19.178 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:18 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:18.212946+0000 mon.smithi077 (mon.0) 307 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:59:19.178 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:18 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:18.213594+0000 mon.smithi077 (mon.0) 308 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T08:59:19.178 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:18 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:18.217399+0000 mon.smithi077 (mon.0) 309 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T08:59:19.178 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:18 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:18.220710+0000 mon.smithi077 (mon.0) 310 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T08:59:19.178 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:18 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:18.223246+0000 mon.smithi077 (mon.0) 311 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:59:19.179 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:18 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:18.224199+0000 mon.smithi077 (mon.0) 312 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:59:19.179 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:18 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:18.224406+0000 mon.smithi077 (mon.0) 313 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:59:19.179 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:18 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:18.224806+0000 mon.smithi077 (mon.0) 314 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:59:19.179 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:18 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:18.832051+0000 mon.smithi141 (mon.1) 6 : audit [DBG] from='client.? 172.21.15.77:0/516060255' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:19.360 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T08:59:20.075 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:19 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:18.217679+0000 mgr.smithi077.amvfyi (mgr.14180) 78 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:20.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:19 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:19.215885+0000 mon.smithi077 (mon.0) 315 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T08:59:20.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:19 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:19.215907+0000 mon.smithi077 (mon.0) 316 : cluster [INF] Cluster is now healthy 2024-01-13T08:59:20.362 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:59:20.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:19 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:18.217679+0000 mgr.smithi077.amvfyi (mgr.14180) 78 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:20.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:19 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:19.215885+0000 mon.smithi077 (mon.0) 315 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T08:59:20.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:19 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:19.215907+0000 mon.smithi077 (mon.0) 316 : cluster [INF] Cluster is now healthy 2024-01-13T08:59:21.248 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:20 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:20.213538+0000 mon.smithi141 (mon.1) 7 : audit [INF] from='client.? 172.21.15.141:0/3261733458' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f5c306df-fb1f-4d34-b6e5-dc76b102a546"}]: dispatch 2024-01-13T08:59:21.248 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:20 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:20.214136+0000 mon.smithi077 (mon.0) 317 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f5c306df-fb1f-4d34-b6e5-dc76b102a546"}]: dispatch 2024-01-13T08:59:21.248 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:20 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:20.218099+0000 mon.smithi077 (mon.0) 318 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f5c306df-fb1f-4d34-b6e5-dc76b102a546"}]': finished 2024-01-13T08:59:21.248 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:20 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:20.218142+0000 mon.smithi077 (mon.0) 319 : cluster [DBG] osdmap e14: 1 total, 0 up, 1 in 2024-01-13T08:59:21.248 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:20 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:20.218240+0000 mon.smithi077 (mon.0) 320 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:59:21.249 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:20 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:20.508041+0000 mon.smithi141 (mon.1) 8 : audit [INF] from='client.? 172.21.15.141:0/1816195599' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:59:21.249 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:20 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:20.508460+0000 mon.smithi077 (mon.0) 321 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:59:21.249 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:20 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:20.511774+0000 mon.smithi077 (mon.0) 322 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:59:21.249 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:20 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:20.511821+0000 mon.smithi077 (mon.0) 323 : cluster [DBG] osdmap e15: 0 total, 0 up, 0 in 2024-01-13T08:59:21.249 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:20 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:20.841408+0000 mon.smithi077 (mon.0) 324 : audit [INF] from='client.? 172.21.15.77:0/760606571' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82b95728-32b1-4c7a-8075-ae04143eaf92"}]: dispatch 2024-01-13T08:59:21.249 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:20 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:20.845057+0000 mon.smithi077 (mon.0) 325 : audit [INF] from='client.? 172.21.15.77:0/760606571' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "82b95728-32b1-4c7a-8075-ae04143eaf92"}]': finished 2024-01-13T08:59:21.249 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:20 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:20.845095+0000 mon.smithi077 (mon.0) 326 : cluster [DBG] osdmap e16: 1 total, 0 up, 1 in 2024-01-13T08:59:21.249 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:20 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:20.845178+0000 mon.smithi077 (mon.0) 327 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:59:21.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:20 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:20.213538+0000 mon.smithi141 (mon.1) 7 : audit [INF] from='client.? 172.21.15.141:0/3261733458' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f5c306df-fb1f-4d34-b6e5-dc76b102a546"}]: dispatch 2024-01-13T08:59:21.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:20 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:20.214136+0000 mon.smithi077 (mon.0) 317 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f5c306df-fb1f-4d34-b6e5-dc76b102a546"}]: dispatch 2024-01-13T08:59:21.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:20 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:20.218099+0000 mon.smithi077 (mon.0) 318 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f5c306df-fb1f-4d34-b6e5-dc76b102a546"}]': finished 2024-01-13T08:59:21.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:20 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:20.218142+0000 mon.smithi077 (mon.0) 319 : cluster [DBG] osdmap e14: 1 total, 0 up, 1 in 2024-01-13T08:59:21.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:20 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:20.218240+0000 mon.smithi077 (mon.0) 320 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:59:21.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:20 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:20.508041+0000 mon.smithi141 (mon.1) 8 : audit [INF] from='client.? 172.21.15.141:0/1816195599' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:59:21.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:20 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:20.508460+0000 mon.smithi077 (mon.0) 321 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:59:21.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:20 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:20.511774+0000 mon.smithi077 (mon.0) 322 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:59:21.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:20 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:20.511821+0000 mon.smithi077 (mon.0) 323 : cluster [DBG] osdmap e15: 0 total, 0 up, 0 in 2024-01-13T08:59:21.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:20 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:20.841408+0000 mon.smithi077 (mon.0) 324 : audit [INF] from='client.? 172.21.15.77:0/760606571' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82b95728-32b1-4c7a-8075-ae04143eaf92"}]: dispatch 2024-01-13T08:59:21.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:20 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:20.845057+0000 mon.smithi077 (mon.0) 325 : audit [INF] from='client.? 172.21.15.77:0/760606571' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "82b95728-32b1-4c7a-8075-ae04143eaf92"}]': finished 2024-01-13T08:59:21.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:20 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:20.845095+0000 mon.smithi077 (mon.0) 326 : cluster [DBG] osdmap e16: 1 total, 0 up, 1 in 2024-01-13T08:59:21.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:20 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:20.845178+0000 mon.smithi077 (mon.0) 327 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:59:21.975 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T08:59:22.290 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:20.217839+0000 mgr.smithi077.amvfyi (mgr.14180) 79 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:22.290 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:59:20.918744+0000 mgr.smithi077.amvfyi (mgr.14180) 80 : cephadm [ERR] executing create_from_spec_one(([('smithi077', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi141', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:59:22.291 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T08:59:22.291 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:59:22.291 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return f(*arg) 2024-01-13T08:59:22.291 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:59:22.291 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:59:22.291 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:59:22.291 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: code, '\n'.join(err))) 2024-01-13T08:59:22.291 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T08:59:22.291 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp3kjf0ycw:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphf8kztos:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:59:22.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:59:22.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:59:22.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:59:22.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new f5c306df-fb1f-4d34-b6e5-dc76b102a546 2024-01-13T08:59:22.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:59:22.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:59:22.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:59:22.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:59:22.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:59:22.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.prepare() 2024-01-13T08:59:22.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:22.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:22.293 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:59:22.293 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:59:22.293 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:59:22.293 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:59:22.293 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:59:22.293 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:59:22.293 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:59:22.293 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T08:59:22.293 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:59:22.293 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T08:59:22.293 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:59:22.294 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:59:22.294 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:59:22.294 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:59:22.294 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:59:22.294 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:59:22.294 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:59:22.294 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:59:22.294 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:59:22.294 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:59:22.294 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T08:59:22.294 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:59:22.294 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:59:22.295 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:59:22.295 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T08:59:22.295 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:22.295 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:22.295 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:59:22.295 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:59:22.295 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:59:22.295 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:59:22.295 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:22.295 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:22.295 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:59:22.296 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:59:22.296 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:59:22.296 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:59:22.296 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:59:22.296 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:59:22.296 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:59:22.296 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:59:22.296 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:22.296 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:22.296 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:59:22.296 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:59:22.296 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:59:22.297 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:59:22.297 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:59:22.297 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T08:59:22.297 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:59:22.297 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: main() 2024-01-13T08:59:22.297 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:59:22.297 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: r = ctx.func(ctx) 2024-01-13T08:59:22.297 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:59:22.297 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:59:22.297 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:59:22.297 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:59:22.298 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:59:22.298 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:59:22.298 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:59:22.298 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:59:22.298 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:59:22.298 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:59:22.298 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:59:22.298 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:59:22.298 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp3kjf0ycw:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphf8kztos:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:59:22.298 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:21.148111+0000 mon.smithi077 (mon.0) 328 : audit [INF] from='client.? 172.21.15.77:0/1566306216' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:59:22.298 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:21.151570+0000 mon.smithi077 (mon.0) 329 : audit [INF] from='client.? 172.21.15.77:0/1566306216' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:59:22.299 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:21.151650+0000 mon.smithi077 (mon.0) 330 : cluster [DBG] osdmap e17: 0 total, 0 up, 0 in 2024-01-13T08:59:22.351 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T08:59:22.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:20.217839+0000 mgr.smithi077.amvfyi (mgr.14180) 79 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:22.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T08:59:20.918744+0000 mgr.smithi077.amvfyi (mgr.14180) 80 : cephadm [ERR] executing create_from_spec_one(([('smithi077', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi141', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:59:22.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T08:59:22.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:59:22.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return f(*arg) 2024-01-13T08:59:22.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:59:22.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:59:22.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:59:22.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: code, '\n'.join(err))) 2024-01-13T08:59:22.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T08:59:22.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp3kjf0ycw:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphf8kztos:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:59:22.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:59:22.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:59:22.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:59:22.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new f5c306df-fb1f-4d34-b6e5-dc76b102a546 2024-01-13T08:59:22.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:59:22.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:59:22.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:59:22.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:59:22.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:59:22.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.prepare() 2024-01-13T08:59:22.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:22.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:22.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:59:22.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:59:22.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:59:22.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:59:22.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:59:22.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:59:22.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:59:22.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T08:59:22.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:59:22.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T08:59:22.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:59:22.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:59:22.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:59:22.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:59:22.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:59:22.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:59:22.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:59:22.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:59:22.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:59:22.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:59:22.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T08:59:22.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:59:22.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:59:22.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:59:22.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T08:59:22.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:22.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:22.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:59:22.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:59:22.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:59:22.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:59:22.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:22.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:22.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:59:22.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:59:22.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:59:22.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:59:22.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:59:22.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:59:22.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:59:22.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:59:22.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:22.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:22.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:59:22.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:59:22.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:59:22.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:59:22.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:59:22.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T08:59:22.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:59:22.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: main() 2024-01-13T08:59:22.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:59:22.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: r = ctx.func(ctx) 2024-01-13T08:59:22.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:59:22.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:59:22.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:59:22.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:59:22.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:59:22.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:59:22.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:59:22.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:59:22.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:59:22.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:59:22.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:59:22.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:59:22.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp3kjf0ycw:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphf8kztos:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:59:22.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:21.148111+0000 mon.smithi077 (mon.0) 328 : audit [INF] from='client.? 172.21.15.77:0/1566306216' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:59:22.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:21.151570+0000 mon.smithi077 (mon.0) 329 : audit [INF] from='client.? 172.21.15.77:0/1566306216' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:59:22.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:21.151650+0000 mon.smithi077 (mon.0) 330 : cluster [DBG] osdmap e17: 0 total, 0 up, 0 in 2024-01-13T08:59:23.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:59:21.676354+0000 mgr.smithi077.amvfyi (mgr.14180) 81 : cephadm [ERR] executing create_from_spec_one(([('smithi077', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi141', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:59:23.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T08:59:23.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:59:23.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return f(*arg) 2024-01-13T08:59:23.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:59:23.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:59:23.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:59:23.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: code, '\n'.join(err))) 2024-01-13T08:59:23.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi077/config 2024-01-13T08:59:23.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptjrpmwrg:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpusqo8cv8:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:59:23.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:59:23.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:59:23.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:59:23.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 82b95728-32b1-4c7a-8075-ae04143eaf92 2024-01-13T08:59:23.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:59:23.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:59:23.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:59:23.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:59:23.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:59:23.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.prepare() 2024-01-13T08:59:23.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:23.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:23.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:59:23.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:59:23.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:59:23.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:59:23.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:59:23.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:59:23.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:59:23.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T08:59:23.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:59:23.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T08:59:23.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:59:23.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:59:23.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:59:23.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:59:23.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:59:23.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:59:23.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:59:23.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:59:23.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:59:23.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:59:23.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T08:59:23.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:59:23.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:59:23.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:59:23.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T08:59:23.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:23.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:23.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:59:23.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:59:23.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:59:23.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:59:23.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:23.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:23.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:59:23.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:59:23.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:59:23.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:59:23.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:59:23.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:59:23.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:59:23.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:59:23.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:23.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:23.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:59:23.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:59:23.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:59:23.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:59:23.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:59:23.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T08:59:23.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:59:23.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: main() 2024-01-13T08:59:23.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:59:23.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: r = ctx.func(ctx) 2024-01-13T08:59:23.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:59:23.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:59:23.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:59:23.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:59:23.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:59:23.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:59:23.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:59:23.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:59:23.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:59:23.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:59:23.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:59:23.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:59:23.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptjrpmwrg:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpusqo8cv8:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:59:23.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T08:59:21.677542+0000 mgr.smithi077.amvfyi (mgr.14180) 82 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T08:59:23.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: service_id: all-available-devices 2024-01-13T08:59:23.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: service_name: osd.all-available-devices 2024-01-13T08:59:23.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: placement: 2024-01-13T08:59:23.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: host_pattern: '*' 2024-01-13T08:59:23.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: spec: 2024-01-13T08:59:23.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: data_devices: 2024-01-13T08:59:23.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: all: true 2024-01-13T08:59:23.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: filter_logic: AND 2024-01-13T08:59:23.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: objectstore: bluestore 2024-01-13T08:59:23.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T08:59:23.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp3kjf0ycw:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphf8kztos:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:59:23.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:59:23.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:59:23.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:59:23.335 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new f5c306df-fb1f-4d34-b6e5-dc76b102a546 2024-01-13T08:59:23.335 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:59:23.335 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:59:23.335 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:59:23.335 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:59:23.335 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:59:23.335 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.prepare() 2024-01-13T08:59:23.335 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:23.335 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:23.335 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:59:23.335 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:59:23.335 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:59:23.335 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:59:23.335 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:59:23.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:59:23.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:59:23.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T08:59:23.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:59:23.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T08:59:23.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:59:23.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:59:23.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:59:23.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:59:23.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:59:23.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:59:23.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:59:23.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:59:23.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:59:23.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:59:23.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T08:59:23.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:59:23.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:59:23.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:59:23.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T08:59:23.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:23.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:23.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:59:23.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:59:23.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:59:23.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:59:23.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:23.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:23.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:59:23.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:59:23.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:59:23.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:59:23.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:59:23.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:59:23.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:59:23.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:59:23.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:23.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:23.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:59:23.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:59:23.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:59:23.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:59:23.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:59:23.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T08:59:23.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:59:23.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: main() 2024-01-13T08:59:23.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:59:23.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: r = ctx.func(ctx) 2024-01-13T08:59:23.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:59:23.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:59:23.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:59:23.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:59:23.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:59:23.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:59:23.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:59:23.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:59:23.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:59:23.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:59:23.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:59:23.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:59:23.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp3kjf0ycw:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphf8kztos:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:59:23.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T08:59:23.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T08:59:23.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: if self._apply_service(spec): 2024-01-13T08:59:23.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T08:59:23.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T08:59:23.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T08:59:23.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T08:59:23.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T08:59:23.341 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T08:59:23.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T08:59:23.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T08:59:23.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T08:59:23.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: raise self._value 2024-01-13T08:59:23.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T08:59:23.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: result = (True, func(*args, **kwds)) 2024-01-13T08:59:23.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T08:59:23.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return list(map(*args)) 2024-01-13T08:59:23.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:59:23.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return f(*arg) 2024-01-13T08:59:23.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:59:23.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:59:23.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:59:23.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: code, '\n'.join(err))) 2024-01-13T08:59:23.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T08:59:23.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp3kjf0ycw:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphf8kztos:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:59:23.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:59:23.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:59:23.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:59:23.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new f5c306df-fb1f-4d34-b6e5-dc76b102a546 2024-01-13T08:59:23.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:59:23.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:59:23.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:59:23.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:59:23.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:59:23.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.prepare() 2024-01-13T08:59:23.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:23.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:23.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:59:23.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:59:23.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:59:23.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:59:23.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:59:23.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:59:23.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:59:23.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T08:59:23.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:59:23.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T08:59:23.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:59:23.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:59:23.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:59:23.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:59:23.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:59:23.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:59:23.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:59:23.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:59:23.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:59:23.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:59:23.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T08:59:23.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:59:23.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:59:23.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:59:23.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T08:59:23.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:23.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:23.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:59:23.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:59:23.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:59:23.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:59:23.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:23.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:23.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:59:23.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:59:23.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:59:23.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:59:23.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:59:23.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:59:23.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:59:23.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:59:23.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:23.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:23.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:59:23.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:59:23.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:59:23.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:59:23.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:59:23.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T08:59:23.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:59:23.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: main() 2024-01-13T08:59:23.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:59:23.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: r = ctx.func(ctx) 2024-01-13T08:59:23.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:59:23.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:59:23.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:59:23.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:59:23.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:59:23.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:59:23.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:59:23.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T08:59:23.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:59:23.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:59:23.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:59:23.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:59:23.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp3kjf0ycw:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphf8kztos:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:59:23.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:21.678079+0000 mgr.smithi077.amvfyi (mgr.14180) 83 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:23.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:21.983562+0000 mon.smithi077 (mon.0) 331 : audit [DBG] from='client.? 172.21.15.77:0/3951745680' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:23.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:22.217319+0000 mon.smithi077 (mon.0) 332 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T08:59:23.352 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:59:23.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T08:59:21.676354+0000 mgr.smithi077.amvfyi (mgr.14180) 81 : cephadm [ERR] executing create_from_spec_one(([('smithi077', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi141', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:59:23.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T08:59:23.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:59:23.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return f(*arg) 2024-01-13T08:59:23.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:59:23.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:59:23.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:59:23.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: code, '\n'.join(err))) 2024-01-13T08:59:23.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi077/config 2024-01-13T08:59:23.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptjrpmwrg:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpusqo8cv8:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:59:23.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:59:23.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:59:23.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:59:23.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 82b95728-32b1-4c7a-8075-ae04143eaf92 2024-01-13T08:59:23.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:59:23.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:59:23.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:59:23.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:59:23.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:59:23.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.prepare() 2024-01-13T08:59:23.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:23.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:23.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:59:23.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:59:23.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:59:23.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:59:23.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:59:23.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:59:23.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:59:23.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T08:59:23.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:59:23.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T08:59:23.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:59:23.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:59:23.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:59:23.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:59:23.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:59:23.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:59:23.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:59:23.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:59:23.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:59:23.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:59:23.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T08:59:23.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:59:23.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:59:23.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:59:23.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T08:59:23.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:23.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:23.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:59:23.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:59:23.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:59:23.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:59:23.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:23.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:23.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:59:23.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:59:23.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:59:23.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:59:23.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:59:23.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:59:23.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:59:23.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:59:23.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:23.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:23.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:59:23.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:59:23.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:59:23.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:59:23.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:59:23.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T08:59:23.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:59:23.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: main() 2024-01-13T08:59:23.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:59:23.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: r = ctx.func(ctx) 2024-01-13T08:59:23.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:59:23.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:59:23.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:59:23.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:59:23.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:59:23.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:59:23.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:59:23.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:59:23.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:59:23.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:59:23.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:59:23.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:59:23.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmptjrpmwrg:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpusqo8cv8:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:59:23.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T08:59:21.677542+0000 mgr.smithi077.amvfyi (mgr.14180) 82 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T08:59:23.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: service_id: all-available-devices 2024-01-13T08:59:23.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: service_name: osd.all-available-devices 2024-01-13T08:59:23.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: placement: 2024-01-13T08:59:23.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: host_pattern: '*' 2024-01-13T08:59:23.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: spec: 2024-01-13T08:59:23.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: data_devices: 2024-01-13T08:59:23.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: all: true 2024-01-13T08:59:23.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: filter_logic: AND 2024-01-13T08:59:23.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: objectstore: bluestore 2024-01-13T08:59:23.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T08:59:23.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp3kjf0ycw:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphf8kztos:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:59:23.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:59:23.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:59:23.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:59:23.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new f5c306df-fb1f-4d34-b6e5-dc76b102a546 2024-01-13T08:59:23.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:59:23.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:59:23.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:59:23.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:59:23.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:59:23.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.prepare() 2024-01-13T08:59:23.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:23.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:23.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:59:23.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:59:23.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:59:23.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:59:23.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:59:23.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:59:23.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:59:23.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T08:59:23.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:59:23.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T08:59:23.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:59:23.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:59:23.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:59:23.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:59:23.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:59:23.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:59:23.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:59:23.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:59:23.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:59:23.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:59:23.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T08:59:23.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:59:23.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:59:23.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:59:23.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T08:59:23.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:23.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:23.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:59:23.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:59:23.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:59:23.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:59:23.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:23.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:23.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:59:23.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:59:23.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:59:23.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:59:23.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:59:23.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:59:23.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:59:23.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:59:23.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:23.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:23.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:59:23.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:59:23.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:59:23.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:59:23.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:59:23.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T08:59:23.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:59:23.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: main() 2024-01-13T08:59:23.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:59:23.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: r = ctx.func(ctx) 2024-01-13T08:59:23.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:59:23.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:59:23.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:59:23.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:59:23.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:59:23.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:59:23.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:59:23.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:59:23.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:59:23.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:59:23.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:59:23.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:59:23.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp3kjf0ycw:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphf8kztos:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:59:23.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T08:59:23.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T08:59:23.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: if self._apply_service(spec): 2024-01-13T08:59:23.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T08:59:23.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T08:59:23.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T08:59:23.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T08:59:23.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T08:59:23.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T08:59:23.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T08:59:23.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T08:59:23.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T08:59:23.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: raise self._value 2024-01-13T08:59:23.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T08:59:23.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: result = (True, func(*args, **kwds)) 2024-01-13T08:59:23.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T08:59:23.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return list(map(*args)) 2024-01-13T08:59:23.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:59:23.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return f(*arg) 2024-01-13T08:59:23.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:59:23.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:59:23.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:59:23.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: code, '\n'.join(err))) 2024-01-13T08:59:23.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T08:59:23.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp3kjf0ycw:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphf8kztos:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:59:23.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:59:23.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:59:23.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:59:23.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new f5c306df-fb1f-4d34-b6e5-dc76b102a546 2024-01-13T08:59:23.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:59:23.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:59:23.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:59:23.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:59:23.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:59:23.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.prepare() 2024-01-13T08:59:23.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:23.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:23.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:59:23.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:59:23.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:59:23.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:59:23.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:59:23.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:59:23.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:59:23.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T08:59:23.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:59:23.397 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T08:59:23.397 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:59:23.397 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:59:23.397 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:59:23.397 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:59:23.397 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:59:23.397 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:59:23.397 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:59:23.397 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:59:23.397 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:59:23.397 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:59:23.397 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T08:59:23.397 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:59:23.397 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:59:23.397 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:59:23.398 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T08:59:23.398 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:23.398 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:23.398 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:59:23.398 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:59:23.398 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:59:23.398 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:59:23.398 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:23.398 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:23.398 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:59:23.398 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:59:23.398 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:59:23.398 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:59:23.398 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:59:23.398 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:59:23.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:59:23.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:59:23.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:59:23.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:59:23.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:59:23.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:59:23.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:59:23.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:59:23.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:59:23.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T08:59:23.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:59:23.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: main() 2024-01-13T08:59:23.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:59:23.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: r = ctx.func(ctx) 2024-01-13T08:59:23.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:59:23.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:59:23.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:59:23.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:59:23.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:59:23.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:59:23.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:59:23.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T08:59:23.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:59:23.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:59:23.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:59:23.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:59:23.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp3kjf0ycw:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphf8kztos:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:59:23.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:21.678079+0000 mgr.smithi077.amvfyi (mgr.14180) 83 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:23.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:21.983562+0000 mon.smithi077 (mon.0) 331 : audit [DBG] from='client.? 172.21.15.77:0/3951745680' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:23.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:22.217319+0000 mon.smithi077 (mon.0) 332 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T08:59:24.735 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T08:59:25.049 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:24 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:23.678624+0000 mgr.smithi077.amvfyi (mgr.14180) 84 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:25.050 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:24 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:24.743125+0000 mon.smithi077 (mon.0) 333 : audit [DBG] from='client.? 172.21.15.77:0/2731934518' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:25.108 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T08:59:25.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:24 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:23.678624+0000 mgr.smithi077.amvfyi (mgr.14180) 84 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:25.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:24 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:24.743125+0000 mon.smithi077 (mon.0) 333 : audit [DBG] from='client.? 172.21.15.77:0/2731934518' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:26.110 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:59:27.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:26 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:25.679077+0000 mgr.smithi077.amvfyi (mgr.14180) 85 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:27.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:26 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:25.679077+0000 mgr.smithi077.amvfyi (mgr.14180) 85 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:27.457 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T08:59:27.818 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T08:59:28.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:27 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:27.464717+0000 mon.smithi077 (mon.0) 334 : audit [DBG] from='client.? 172.21.15.77:0/271305638' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:28.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:27 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:27.464717+0000 mon.smithi077 (mon.0) 334 : audit [DBG] from='client.? 172.21.15.77:0/271305638' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:28.819 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:59:29.153 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:27.679582+0000 mgr.smithi077.amvfyi (mgr.14180) 86 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:29.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:27.679582+0000 mgr.smithi077.amvfyi (mgr.14180) 86 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:30.214 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T08:59:30.592 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T08:59:31.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:30 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:29.680214+0000 mgr.smithi077.amvfyi (mgr.14180) 87 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:31.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:30 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:30.222425+0000 mon.smithi077 (mon.0) 335 : audit [DBG] from='client.? 172.21.15.77:0/2329849614' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:31.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:30 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:29.680214+0000 mgr.smithi077.amvfyi (mgr.14180) 87 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:31.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:30 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:30.222425+0000 mon.smithi077 (mon.0) 335 : audit [DBG] from='client.? 172.21.15.77:0/2329849614' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:31.593 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:59:32.922 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T08:59:33.265 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:31.680663+0000 mgr.smithi077.amvfyi (mgr.14180) 88 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:33.265 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:32.930127+0000 mon.smithi077 (mon.0) 336 : audit [DBG] from='client.? 172.21.15.77:0/1000189394' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:33.340 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T08:59:33.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:31.680663+0000 mgr.smithi077.amvfyi (mgr.14180) 88 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:33.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:32.930127+0000 mon.smithi077 (mon.0) 336 : audit [DBG] from='client.? 172.21.15.77:0/1000189394' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:34.341 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:59:35.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:35 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:33.681066+0000 mgr.smithi077.amvfyi (mgr.14180) 89 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:35.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:35 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:33.681066+0000 mgr.smithi077.amvfyi (mgr.14180) 89 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:35.721 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T08:59:36.108 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T08:59:36.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:36 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:35.729192+0000 mon.smithi077 (mon.0) 337 : audit [DBG] from='client.? 172.21.15.77:0/1338407618' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:36.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:36 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:35.729192+0000 mon.smithi077 (mon.0) 337 : audit [DBG] from='client.? 172.21.15.77:0/1338407618' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:37.109 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:59:37.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:35.681482+0000 mgr.smithi077.amvfyi (mgr.14180) 90 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:37.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:35.681482+0000 mgr.smithi077.amvfyi (mgr.14180) 90 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:38.458 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T08:59:38.848 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T08:59:39.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:39 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:37.682080+0000 mgr.smithi077.amvfyi (mgr.14180) 91 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:39.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:39 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:38.465164+0000 mon.smithi077 (mon.0) 338 : audit [DBG] from='client.? 172.21.15.77:0/1835270056' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:39.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:39 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:37.682080+0000 mgr.smithi077.amvfyi (mgr.14180) 91 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:39.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:39 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:38.465164+0000 mon.smithi077 (mon.0) 338 : audit [DBG] from='client.? 172.21.15.77:0/1835270056' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:39.850 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:59:41.213 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T08:59:41.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:41 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:39.682635+0000 mgr.smithi077.amvfyi (mgr.14180) 92 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:41.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:41 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:39.682635+0000 mgr.smithi077.amvfyi (mgr.14180) 92 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:41.589 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T08:59:42.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:42 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:41.220512+0000 mon.smithi077 (mon.0) 339 : audit [DBG] from='client.? 172.21.15.77:0/3614333910' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:42.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:42 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:41.220512+0000 mon.smithi077 (mon.0) 339 : audit [DBG] from='client.? 172.21.15.77:0/3614333910' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:42.591 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:59:43.252 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:43 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:41.683257+0000 mgr.smithi077.amvfyi (mgr.14180) 93 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:43.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:43 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:41.683257+0000 mgr.smithi077.amvfyi (mgr.14180) 93 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:43.938 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T08:59:44.263 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:44 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:43.945876+0000 mon.smithi077 (mon.0) 340 : audit [DBG] from='client.? 172.21.15.77:0/2065770649' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:44.306 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T08:59:44.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:44 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:43.945876+0000 mon.smithi077 (mon.0) 340 : audit [DBG] from='client.? 172.21.15.77:0/2065770649' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:45.308 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:59:45.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:45 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:43.683609+0000 mgr.smithi077.amvfyi (mgr.14180) 94 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:45.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:45 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:43.683609+0000 mgr.smithi077.amvfyi (mgr.14180) 94 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:46.678 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T08:59:47.098 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T08:59:47.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:47 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:45.683948+0000 mgr.smithi077.amvfyi (mgr.14180) 95 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:47.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:47 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:46.685874+0000 mon.smithi077 (mon.0) 341 : audit [DBG] from='client.? 172.21.15.77:0/23458386' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:47.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:47 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:45.683948+0000 mgr.smithi077.amvfyi (mgr.14180) 95 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:47.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:47 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:46.685874+0000 mon.smithi077 (mon.0) 341 : audit [DBG] from='client.? 172.21.15.77:0/23458386' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:48.099 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:59:49.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:49 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:47.684347+0000 mgr.smithi077.amvfyi (mgr.14180) 96 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:49.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:49 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:47.684347+0000 mgr.smithi077.amvfyi (mgr.14180) 96 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:49.470 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T08:59:49.864 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T08:59:50.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:50 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:49.478325+0000 mon.smithi077 (mon.0) 342 : audit [DBG] from='client.? 172.21.15.77:0/2371441216' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:50.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:50 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:49.478325+0000 mon.smithi077 (mon.0) 342 : audit [DBG] from='client.? 172.21.15.77:0/2371441216' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:50.865 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:59:51.224 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:51 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:49.684750+0000 mgr.smithi077.amvfyi (mgr.14180) 97 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:51.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:51 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:49.684750+0000 mgr.smithi077.amvfyi (mgr.14180) 97 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:52.221 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T08:59:52.630 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T08:59:53.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:51.685260+0000 mgr.smithi077.amvfyi (mgr.14180) 98 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:53.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:52.228565+0000 mon.smithi077 (mon.0) 343 : audit [DBG] from='client.? 172.21.15.77:0/1413973713' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:53.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:51.685260+0000 mgr.smithi077.amvfyi (mgr.14180) 98 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:53.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:52.228565+0000 mon.smithi077 (mon.0) 343 : audit [DBG] from='client.? 172.21.15.77:0/1413973713' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:53.631 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:59:54.969 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T08:59:55.321 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:55 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:53.685683+0000 mgr.smithi077.amvfyi (mgr.14180) 99 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:55.321 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:55 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:54.976729+0000 mon.smithi077 (mon.0) 344 : audit [DBG] from='client.? 172.21.15.77:0/263373330' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:55.355 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T08:59:55.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:55 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:53.685683+0000 mgr.smithi077.amvfyi (mgr.14180) 99 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:55.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:55 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:54.976729+0000 mon.smithi077 (mon.0) 344 : audit [DBG] from='client.? 172.21.15.77:0/263373330' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:56.356 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:59:57.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:57 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:55.686366+0000 mgr.smithi077.amvfyi (mgr.14180) 100 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:57.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:57 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:55.686366+0000 mgr.smithi077.amvfyi (mgr.14180) 100 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:57.625 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T08:59:57.997 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T08:59:58.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:58 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T08:59:57.633274+0000 mon.smithi077 (mon.0) 345 : audit [DBG] from='client.? 172.21.15.77:0/2859741622' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:58.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:58 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T08:59:57.633274+0000 mon.smithi077 (mon.0) 345 : audit [DBG] from='client.? 172.21.15.77:0/2859741622' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:59:58.998 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:59:59.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 08:59:59 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:57.686801+0000 mgr.smithi077.amvfyi (mgr.14180) 101 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:59:59.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 08:59:59 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:57.686801+0000 mgr.smithi077.amvfyi (mgr.14180) 101 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:00.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000139+0000 mon.smithi077 (mon.0) 346 : cluster [WRN] Health detail: HEALTH_WARN Failed to apply 1 service(s): osd.all-available-devices 2024-01-13T09:00:00.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000168+0000 mon.smithi077 (mon.0) 347 : cluster [WRN] [WRN] CEPHADM_APPLY_SPEC_FAIL: Failed to apply 1 service(s): osd.all-available-devices 2024-01-13T09:00:00.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000183+0000 mon.smithi077 (mon.0) 348 : cluster [WRN] osd.all-available-devices: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T09:00:00.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000208+0000 mon.smithi077 (mon.0) 349 : cluster [WRN] Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp3kjf0ycw:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphf8kztos:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:00:00.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000224+0000 mon.smithi077 (mon.0) 350 : cluster [WRN] /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:00:00.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000249+0000 mon.smithi077 (mon.0) 351 : cluster [WRN] /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:00:00.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000262+0000 mon.smithi077 (mon.0) 352 : cluster [WRN] /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:00:00.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000277+0000 mon.smithi077 (mon.0) 353 : cluster [WRN] /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new f5c306df-fb1f-4d34-b6e5-dc76b102a546 2024-01-13T09:00:00.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000291+0000 mon.smithi077 (mon.0) 354 : cluster [WRN] /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:00:00.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000306+0000 mon.smithi077 (mon.0) 355 : cluster [WRN] /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:00:00.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000320+0000 mon.smithi077 (mon.0) 356 : cluster [WRN] /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:00:00.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000335+0000 mon.smithi077 (mon.0) 357 : cluster [WRN] /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:00:00.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000351+0000 mon.smithi077 (mon.0) 358 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:00:00.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000368+0000 mon.smithi077 (mon.0) 359 : cluster [WRN] /bin/podman: stderr self.prepare() 2024-01-13T09:00:00.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000380+0000 mon.smithi077 (mon.0) 360 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:00.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000390+0000 mon.smithi077 (mon.0) 361 : cluster [WRN] /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:00.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000399+0000 mon.smithi077 (mon.0) 362 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:00:00.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000407+0000 mon.smithi077 (mon.0) 363 : cluster [WRN] /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:00:00.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000420+0000 mon.smithi077 (mon.0) 364 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:00:00.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000429+0000 mon.smithi077 (mon.0) 365 : cluster [WRN] /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:00:00.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000438+0000 mon.smithi077 (mon.0) 366 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:00:00.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000446+0000 mon.smithi077 (mon.0) 367 : cluster [WRN] /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:00:00.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000456+0000 mon.smithi077 (mon.0) 368 : cluster [WRN] /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:00:00.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000466+0000 mon.smithi077 (mon.0) 369 : cluster [WRN] /bin/podman: stderr 2024-01-13T09:00:00.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000474+0000 mon.smithi077 (mon.0) 370 : cluster [WRN] /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:00:00.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000483+0000 mon.smithi077 (mon.0) 371 : cluster [WRN] /bin/podman: stderr 2024-01-13T09:00:00.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000492+0000 mon.smithi077 (mon.0) 372 : cluster [WRN] /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:00:00.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000501+0000 mon.smithi077 (mon.0) 373 : cluster [WRN] /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:00:00.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000515+0000 mon.smithi077 (mon.0) 374 : cluster [WRN] /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:00:00.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000528+0000 mon.smithi077 (mon.0) 375 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:00:00.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000538+0000 mon.smithi077 (mon.0) 376 : cluster [WRN] /bin/podman: stderr self.main(self.argv) 2024-01-13T09:00:00.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000546+0000 mon.smithi077 (mon.0) 377 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:00:00.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000558+0000 mon.smithi077 (mon.0) 378 : cluster [WRN] /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:00:00.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000569+0000 mon.smithi077 (mon.0) 379 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:00:00.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000584+0000 mon.smithi077 (mon.0) 380 : cluster [WRN] /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:00:00.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000600+0000 mon.smithi077 (mon.0) 381 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:00:00.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000621+0000 mon.smithi077 (mon.0) 382 : cluster [WRN] /bin/podman: stderr instance.main() 2024-01-13T09:00:00.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000636+0000 mon.smithi077 (mon.0) 383 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:00:00.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000650+0000 mon.smithi077 (mon.0) 384 : cluster [WRN] /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:00:00.330 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000664+0000 mon.smithi077 (mon.0) 385 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:00:00.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000678+0000 mon.smithi077 (mon.0) 386 : cluster [WRN] /bin/podman: stderr instance.main() 2024-01-13T09:00:00.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000693+0000 mon.smithi077 (mon.0) 387 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:00.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000708+0000 mon.smithi077 (mon.0) 388 : cluster [WRN] /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:00.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000722+0000 mon.smithi077 (mon.0) 389 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:00:00.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000736+0000 mon.smithi077 (mon.0) 390 : cluster [WRN] /bin/podman: stderr self._execute(plan) 2024-01-13T09:00:00.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000751+0000 mon.smithi077 (mon.0) 391 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:00:00.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000762+0000 mon.smithi077 (mon.0) 392 : cluster [WRN] /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:00:00.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000772+0000 mon.smithi077 (mon.0) 393 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:00.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000785+0000 mon.smithi077 (mon.0) 394 : cluster [WRN] /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:00.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000794+0000 mon.smithi077 (mon.0) 395 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:00:00.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000803+0000 mon.smithi077 (mon.0) 396 : cluster [WRN] /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:00:00.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000810+0000 mon.smithi077 (mon.0) 397 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:00:00.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000817+0000 mon.smithi077 (mon.0) 398 : cluster [WRN] /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:00:00.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000834+0000 mon.smithi077 (mon.0) 399 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:00:00.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000853+0000 mon.smithi077 (mon.0) 400 : cluster [WRN] /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:00:00.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000862+0000 mon.smithi077 (mon.0) 401 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:00:00.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000872+0000 mon.smithi077 (mon.0) 402 : cluster [WRN] /bin/podman: stderr self.zap_osd() 2024-01-13T09:00:00.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000889+0000 mon.smithi077 (mon.0) 403 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:00.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000900+0000 mon.smithi077 (mon.0) 404 : cluster [WRN] /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:00.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000912+0000 mon.smithi077 (mon.0) 405 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:00:00.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000922+0000 mon.smithi077 (mon.0) 406 : cluster [WRN] /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:00:00.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000931+0000 mon.smithi077 (mon.0) 407 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:00:00.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000943+0000 mon.smithi077 (mon.0) 408 : cluster [WRN] /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:00:00.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000956+0000 mon.smithi077 (mon.0) 409 : cluster [WRN] /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:00:00.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000965+0000 mon.smithi077 (mon.0) 410 : cluster [WRN] Traceback (most recent call last): 2024-01-13T09:00:00.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000984+0000 mon.smithi077 (mon.0) 411 : cluster [WRN] File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:00:00.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.000993+0000 mon.smithi077 (mon.0) 412 : cluster [WRN] main() 2024-01-13T09:00:00.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.001001+0000 mon.smithi077 (mon.0) 413 : cluster [WRN] File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:00:00.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.001010+0000 mon.smithi077 (mon.0) 414 : cluster [WRN] r = ctx.func(ctx) 2024-01-13T09:00:00.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.001022+0000 mon.smithi077 (mon.0) 415 : cluster [WRN] File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:00:00.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.001032+0000 mon.smithi077 (mon.0) 416 : cluster [WRN] return func(ctx) 2024-01-13T09:00:00.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.001041+0000 mon.smithi077 (mon.0) 417 : cluster [WRN] File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:00:00.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.001056+0000 mon.smithi077 (mon.0) 418 : cluster [WRN] return func(ctx) 2024-01-13T09:00:00.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.001064+0000 mon.smithi077 (mon.0) 419 : cluster [WRN] File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:00:00.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.001073+0000 mon.smithi077 (mon.0) 420 : cluster [WRN] return func(ctx) 2024-01-13T09:00:00.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.001082+0000 mon.smithi077 (mon.0) 421 : cluster [WRN] File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:00:00.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.001091+0000 mon.smithi077 (mon.0) 422 : cluster [WRN] return func(ctx) 2024-01-13T09:00:00.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.001099+0000 mon.smithi077 (mon.0) 423 : cluster [WRN] File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:00:00.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.001109+0000 mon.smithi077 (mon.0) 424 : cluster [WRN] out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:00:00.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.001119+0000 mon.smithi077 (mon.0) 425 : cluster [WRN] File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:00:00.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.001130+0000 mon.smithi077 (mon.0) 426 : cluster [WRN] raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:00:00.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:00.001140+0000 mon.smithi077 (mon.0) 427 : cluster [WRN] RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp3kjf0ycw:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphf8kztos:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:00:00.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000139+0000 mon.smithi077 (mon.0) 346 : cluster [WRN] Health detail: HEALTH_WARN Failed to apply 1 service(s): osd.all-available-devices 2024-01-13T09:00:00.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000168+0000 mon.smithi077 (mon.0) 347 : cluster [WRN] [WRN] CEPHADM_APPLY_SPEC_FAIL: Failed to apply 1 service(s): osd.all-available-devices 2024-01-13T09:00:00.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000183+0000 mon.smithi077 (mon.0) 348 : cluster [WRN] osd.all-available-devices: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T09:00:00.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000208+0000 mon.smithi077 (mon.0) 349 : cluster [WRN] Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp3kjf0ycw:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphf8kztos:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:00:00.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000224+0000 mon.smithi077 (mon.0) 350 : cluster [WRN] /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:00:00.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000249+0000 mon.smithi077 (mon.0) 351 : cluster [WRN] /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:00:00.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000262+0000 mon.smithi077 (mon.0) 352 : cluster [WRN] /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:00:00.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000277+0000 mon.smithi077 (mon.0) 353 : cluster [WRN] /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new f5c306df-fb1f-4d34-b6e5-dc76b102a546 2024-01-13T09:00:00.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000291+0000 mon.smithi077 (mon.0) 354 : cluster [WRN] /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:00:00.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000306+0000 mon.smithi077 (mon.0) 355 : cluster [WRN] /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:00:00.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000320+0000 mon.smithi077 (mon.0) 356 : cluster [WRN] /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:00:00.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000335+0000 mon.smithi077 (mon.0) 357 : cluster [WRN] /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:00:00.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000351+0000 mon.smithi077 (mon.0) 358 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:00:00.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000368+0000 mon.smithi077 (mon.0) 359 : cluster [WRN] /bin/podman: stderr self.prepare() 2024-01-13T09:00:00.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000380+0000 mon.smithi077 (mon.0) 360 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:00.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000390+0000 mon.smithi077 (mon.0) 361 : cluster [WRN] /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:00.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000399+0000 mon.smithi077 (mon.0) 362 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:00:00.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000407+0000 mon.smithi077 (mon.0) 363 : cluster [WRN] /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:00:00.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000420+0000 mon.smithi077 (mon.0) 364 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:00:00.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000429+0000 mon.smithi077 (mon.0) 365 : cluster [WRN] /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:00:00.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000438+0000 mon.smithi077 (mon.0) 366 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:00:00.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000446+0000 mon.smithi077 (mon.0) 367 : cluster [WRN] /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:00:00.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000456+0000 mon.smithi077 (mon.0) 368 : cluster [WRN] /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:00:00.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000466+0000 mon.smithi077 (mon.0) 369 : cluster [WRN] /bin/podman: stderr 2024-01-13T09:00:00.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000474+0000 mon.smithi077 (mon.0) 370 : cluster [WRN] /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:00:00.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000483+0000 mon.smithi077 (mon.0) 371 : cluster [WRN] /bin/podman: stderr 2024-01-13T09:00:00.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000492+0000 mon.smithi077 (mon.0) 372 : cluster [WRN] /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:00:00.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000501+0000 mon.smithi077 (mon.0) 373 : cluster [WRN] /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:00:00.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000515+0000 mon.smithi077 (mon.0) 374 : cluster [WRN] /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:00:00.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000528+0000 mon.smithi077 (mon.0) 375 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:00:00.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000538+0000 mon.smithi077 (mon.0) 376 : cluster [WRN] /bin/podman: stderr self.main(self.argv) 2024-01-13T09:00:00.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000546+0000 mon.smithi077 (mon.0) 377 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:00:00.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000558+0000 mon.smithi077 (mon.0) 378 : cluster [WRN] /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:00:00.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000569+0000 mon.smithi077 (mon.0) 379 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:00:00.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000584+0000 mon.smithi077 (mon.0) 380 : cluster [WRN] /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:00:00.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000600+0000 mon.smithi077 (mon.0) 381 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:00:00.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000621+0000 mon.smithi077 (mon.0) 382 : cluster [WRN] /bin/podman: stderr instance.main() 2024-01-13T09:00:00.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000636+0000 mon.smithi077 (mon.0) 383 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:00:00.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000650+0000 mon.smithi077 (mon.0) 384 : cluster [WRN] /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:00:00.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000664+0000 mon.smithi077 (mon.0) 385 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:00:00.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000678+0000 mon.smithi077 (mon.0) 386 : cluster [WRN] /bin/podman: stderr instance.main() 2024-01-13T09:00:00.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000693+0000 mon.smithi077 (mon.0) 387 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:00.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000708+0000 mon.smithi077 (mon.0) 388 : cluster [WRN] /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:00.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000722+0000 mon.smithi077 (mon.0) 389 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:00:00.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000736+0000 mon.smithi077 (mon.0) 390 : cluster [WRN] /bin/podman: stderr self._execute(plan) 2024-01-13T09:00:00.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000751+0000 mon.smithi077 (mon.0) 391 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:00:00.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000762+0000 mon.smithi077 (mon.0) 392 : cluster [WRN] /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:00:00.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000772+0000 mon.smithi077 (mon.0) 393 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:00.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000785+0000 mon.smithi077 (mon.0) 394 : cluster [WRN] /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:00.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000794+0000 mon.smithi077 (mon.0) 395 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:00:00.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000803+0000 mon.smithi077 (mon.0) 396 : cluster [WRN] /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:00:00.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000810+0000 mon.smithi077 (mon.0) 397 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:00:00.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000817+0000 mon.smithi077 (mon.0) 398 : cluster [WRN] /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:00:00.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000834+0000 mon.smithi077 (mon.0) 399 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:00:00.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000853+0000 mon.smithi077 (mon.0) 400 : cluster [WRN] /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:00:00.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000862+0000 mon.smithi077 (mon.0) 401 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:00:00.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000872+0000 mon.smithi077 (mon.0) 402 : cluster [WRN] /bin/podman: stderr self.zap_osd() 2024-01-13T09:00:00.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000889+0000 mon.smithi077 (mon.0) 403 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:00.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000900+0000 mon.smithi077 (mon.0) 404 : cluster [WRN] /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:00.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000912+0000 mon.smithi077 (mon.0) 405 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:00:00.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000922+0000 mon.smithi077 (mon.0) 406 : cluster [WRN] /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:00:00.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000931+0000 mon.smithi077 (mon.0) 407 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:00:00.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000943+0000 mon.smithi077 (mon.0) 408 : cluster [WRN] /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:00:00.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000956+0000 mon.smithi077 (mon.0) 409 : cluster [WRN] /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:00:00.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000965+0000 mon.smithi077 (mon.0) 410 : cluster [WRN] Traceback (most recent call last): 2024-01-13T09:00:00.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000984+0000 mon.smithi077 (mon.0) 411 : cluster [WRN] File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:00:00.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.000993+0000 mon.smithi077 (mon.0) 412 : cluster [WRN] main() 2024-01-13T09:00:00.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.001001+0000 mon.smithi077 (mon.0) 413 : cluster [WRN] File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:00:00.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.001010+0000 mon.smithi077 (mon.0) 414 : cluster [WRN] r = ctx.func(ctx) 2024-01-13T09:00:00.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.001022+0000 mon.smithi077 (mon.0) 415 : cluster [WRN] File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:00:00.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.001032+0000 mon.smithi077 (mon.0) 416 : cluster [WRN] return func(ctx) 2024-01-13T09:00:00.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.001041+0000 mon.smithi077 (mon.0) 417 : cluster [WRN] File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:00:00.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.001056+0000 mon.smithi077 (mon.0) 418 : cluster [WRN] return func(ctx) 2024-01-13T09:00:00.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.001064+0000 mon.smithi077 (mon.0) 419 : cluster [WRN] File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:00:00.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.001073+0000 mon.smithi077 (mon.0) 420 : cluster [WRN] return func(ctx) 2024-01-13T09:00:00.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.001082+0000 mon.smithi077 (mon.0) 421 : cluster [WRN] File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:00:00.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.001091+0000 mon.smithi077 (mon.0) 422 : cluster [WRN] return func(ctx) 2024-01-13T09:00:00.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.001099+0000 mon.smithi077 (mon.0) 423 : cluster [WRN] File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:00:00.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.001109+0000 mon.smithi077 (mon.0) 424 : cluster [WRN] out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:00:00.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.001119+0000 mon.smithi077 (mon.0) 425 : cluster [WRN] File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:00:00.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.001130+0000 mon.smithi077 (mon.0) 426 : cluster [WRN] raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:00:00.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:00.001140+0000 mon.smithi077 (mon.0) 427 : cluster [WRN] RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp3kjf0ycw:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphf8kztos:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:00:00.386 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:00:00.788 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:00:01.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:01 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T08:59:59.687415+0000 mgr.smithi077.amvfyi (mgr.14180) 102 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:01.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:01 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:00.389353+0000 mon.smithi077 (mon.0) 428 : audit [DBG] from='client.? 172.21.15.77:0/401462711' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:01.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:01 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T08:59:59.687415+0000 mgr.smithi077.amvfyi (mgr.14180) 102 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:01.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:01 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:00.389353+0000 mon.smithi077 (mon.0) 428 : audit [DBG] from='client.? 172.21.15.77:0/401462711' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:01.790 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:00:03.104 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:00:03.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:03 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:01.687767+0000 mgr.smithi077.amvfyi (mgr.14180) 103 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:03.462 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:03 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:01.687767+0000 mgr.smithi077.amvfyi (mgr.14180) 103 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:03.505 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:00:04.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:04 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:03.112112+0000 mon.smithi077 (mon.0) 429 : audit [DBG] from='client.? 172.21.15.77:0/1394377172' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:04.506 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:00:04.539 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:04 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:03.112112+0000 mon.smithi077 (mon.0) 429 : audit [DBG] from='client.? 172.21.15.77:0/1394377172' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:05.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:05 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:03.688123+0000 mgr.smithi077.amvfyi (mgr.14180) 104 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:05.475 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:05 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:03.688123+0000 mgr.smithi077.amvfyi (mgr.14180) 104 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:05.817 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:00:06.197 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:00:06.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:06 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:05.825332+0000 mon.smithi077 (mon.0) 430 : audit [DBG] from='client.? 172.21.15.77:0/1325268572' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:06.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:06 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:05.825332+0000 mon.smithi077 (mon.0) 430 : audit [DBG] from='client.? 172.21.15.77:0/1325268572' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:07.198 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:00:07.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:07 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:05.688501+0000 mgr.smithi077.amvfyi (mgr.14180) 105 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:07.572 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:07 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:05.688501+0000 mgr.smithi077.amvfyi (mgr.14180) 105 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:08.577 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:00:08.987 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:00:09.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:09 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:07.689089+0000 mgr.smithi077.amvfyi (mgr.14180) 106 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:09.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:09 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:08.584830+0000 mon.smithi077 (mon.0) 431 : audit [DBG] from='client.? 172.21.15.77:0/4143769736' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:09.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:09 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:07.689089+0000 mgr.smithi077.amvfyi (mgr.14180) 106 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:09.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:09 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:08.584830+0000 mon.smithi077 (mon.0) 431 : audit [DBG] from='client.? 172.21.15.77:0/4143769736' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:09.989 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:00:11.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:11 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:09.689480+0000 mgr.smithi077.amvfyi (mgr.14180) 107 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:11.333 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:00:11.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:11 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:09.689480+0000 mgr.smithi077.amvfyi (mgr.14180) 107 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:11.711 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:00:12.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:12 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:11.341150+0000 mon.smithi077 (mon.0) 432 : audit [DBG] from='client.? 172.21.15.77:0/1680065779' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:12.575 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:12 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:11.341150+0000 mon.smithi077 (mon.0) 432 : audit [DBG] from='client.? 172.21.15.77:0/1680065779' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:12.712 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:00:13.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:13 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:11.689839+0000 mgr.smithi077.amvfyi (mgr.14180) 108 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:13.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:13 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:11.689839+0000 mgr.smithi077.amvfyi (mgr.14180) 108 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:14.043 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:00:14.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:14.050374+0000 mon.smithi077 (mon.0) 433 : audit [DBG] from='client.? 172.21.15.77:0/3774444773' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:14.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:14.050374+0000 mon.smithi077 (mon.0) 433 : audit [DBG] from='client.? 172.21.15.77:0/3774444773' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:14.418 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:00:15.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:13.690273+0000 mgr.smithi077.amvfyi (mgr.14180) 109 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:15.420 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:00:15.450 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:13.690273+0000 mgr.smithi077.amvfyi (mgr.14180) 109 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:16.731 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:00:17.138 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:00:17.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:17 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:15.690679+0000 mgr.smithi077.amvfyi (mgr.14180) 110 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:17.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:17 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:16.739150+0000 mon.smithi077 (mon.0) 434 : audit [DBG] from='client.? 172.21.15.77:0/2764051787' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:17.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:17 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:15.690679+0000 mgr.smithi077.amvfyi (mgr.14180) 110 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:17.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:17 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:16.739150+0000 mon.smithi077 (mon.0) 434 : audit [DBG] from='client.? 172.21.15.77:0/2764051787' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:18.139 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:00:19.499 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:00:19.528 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:19 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:17.691181+0000 mgr.smithi077.amvfyi (mgr.14180) 111 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:19.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:19 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:17.691181+0000 mgr.smithi077.amvfyi (mgr.14180) 111 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:19.895 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:00:20.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:20 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:19.507253+0000 mon.smithi077 (mon.0) 435 : audit [DBG] from='client.? 172.21.15.77:0/3976057405' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:20.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:20 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:19.507253+0000 mon.smithi077 (mon.0) 435 : audit [DBG] from='client.? 172.21.15.77:0/3976057405' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:20.896 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:00:21.271 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:19.691752+0000 mgr.smithi077.amvfyi (mgr.14180) 112 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:21.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:19.691752+0000 mgr.smithi077.amvfyi (mgr.14180) 112 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:22.295 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:00:22.321 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:21.679059+0000 mon.smithi077 (mon.0) 436 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T09:00:22.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:21.679059+0000 mon.smithi077 (mon.0) 436 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T09:00:22.780 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:00:23.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:23 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:21.692143+0000 mgr.smithi077.amvfyi (mgr.14180) 113 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:23.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:23 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:22.302585+0000 mon.smithi077 (mon.0) 437 : audit [DBG] from='client.? 172.21.15.77:0/3787687439' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:23.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:23 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:22.841298+0000 mon.smithi077 (mon.0) 438 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T09:00:23.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:23 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:21.692143+0000 mgr.smithi077.amvfyi (mgr.14180) 113 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:23.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:23 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:22.302585+0000 mon.smithi077 (mon.0) 437 : audit [DBG] from='client.? 172.21.15.77:0/3787687439' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:23.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:23 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:22.841298+0000 mon.smithi077 (mon.0) 438 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T09:00:23.781 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:00:25.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:25 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:23.692519+0000 mgr.smithi077.amvfyi (mgr.14180) 114 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:25.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:25 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:24.783334+0000 mon.smithi077 (mon.0) 439 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T09:00:25.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:25 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:25.111023+0000 mon.smithi077 (mon.0) 440 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:00:25.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:25 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:25.111666+0000 mon.smithi077 (mon.0) 441 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T09:00:25.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:25 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:25.115707+0000 mon.smithi077 (mon.0) 442 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T09:00:25.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:25 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:25.119448+0000 mon.smithi077 (mon.0) 443 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T09:00:25.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:25 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:25.122788+0000 mon.smithi077 (mon.0) 444 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:00:25.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:25 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:25.123541+0000 mon.smithi077 (mon.0) 445 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:00:25.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:25 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:25.123746+0000 mon.smithi077 (mon.0) 446 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:00:25.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:25 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:25.124202+0000 mon.smithi077 (mon.0) 447 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:00:25.490 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:25 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:23.692519+0000 mgr.smithi077.amvfyi (mgr.14180) 114 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:25.490 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:25 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:24.783334+0000 mon.smithi077 (mon.0) 439 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T09:00:25.490 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:25 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:25.111023+0000 mon.smithi077 (mon.0) 440 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:00:25.490 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:25 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:25.111666+0000 mon.smithi077 (mon.0) 441 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T09:00:25.490 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:25 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:25.115707+0000 mon.smithi077 (mon.0) 442 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T09:00:25.491 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:25 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:25.119448+0000 mon.smithi077 (mon.0) 443 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T09:00:25.491 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:25 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:25.122788+0000 mon.smithi077 (mon.0) 444 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:00:25.491 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:25 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:25.123541+0000 mon.smithi077 (mon.0) 445 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:00:25.491 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:25 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:25.123746+0000 mon.smithi077 (mon.0) 446 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:00:25.491 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:25 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:25.124202+0000 mon.smithi077 (mon.0) 447 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:00:25.507 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:00:26.238 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:00:26.575 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:26 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:25.116004+0000 mgr.smithi077.amvfyi (mgr.14180) 115 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:26.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:26 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:25.515195+0000 mon.smithi077 (mon.0) 448 : audit [DBG] from='client.? 172.21.15.77:0/443237668' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:26.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:26 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:26.114104+0000 mon.smithi077 (mon.0) 449 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T09:00:26.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:26 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:26.114180+0000 mon.smithi077 (mon.0) 450 : cluster [INF] Cluster is now healthy 2024-01-13T09:00:26.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:26 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:25.116004+0000 mgr.smithi077.amvfyi (mgr.14180) 115 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:26.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:26 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:25.515195+0000 mon.smithi077 (mon.0) 448 : audit [DBG] from='client.? 172.21.15.77:0/443237668' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:26.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:26 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:26.114104+0000 mon.smithi077 (mon.0) 449 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T09:00:26.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:26 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:26.114180+0000 mon.smithi077 (mon.0) 450 : cluster [INF] Cluster is now healthy 2024-01-13T09:00:27.238 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:00:28.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:27 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:27.116246+0000 mgr.smithi077.amvfyi (mgr.14180) 116 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:28.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:27 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:27.228250+0000 mon.smithi141 (mon.1) 9 : audit [INF] from='client.? 172.21.15.141:0/278813858' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "276e41e0-b262-4fdc-8ca7-421e3fb7629f"}]: dispatch 2024-01-13T09:00:28.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:27 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:27.228599+0000 mon.smithi077 (mon.0) 451 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "276e41e0-b262-4fdc-8ca7-421e3fb7629f"}]: dispatch 2024-01-13T09:00:28.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:27 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:27.236464+0000 mon.smithi077 (mon.0) 452 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "276e41e0-b262-4fdc-8ca7-421e3fb7629f"}]': finished 2024-01-13T09:00:28.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:27 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:27.236507+0000 mon.smithi077 (mon.0) 453 : cluster [DBG] osdmap e18: 1 total, 0 up, 1 in 2024-01-13T09:00:28.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:27 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:27.236592+0000 mon.smithi077 (mon.0) 454 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:00:28.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:27 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:27.406799+0000 mon.smithi077 (mon.0) 455 : audit [INF] from='client.? 172.21.15.77:0/4108059452' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "435afe84-60df-457c-8c7f-1fed94ed7f02"}]: dispatch 2024-01-13T09:00:28.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:27 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:27.414702+0000 mon.smithi077 (mon.0) 456 : audit [INF] from='client.? 172.21.15.77:0/4108059452' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "435afe84-60df-457c-8c7f-1fed94ed7f02"}]': finished 2024-01-13T09:00:28.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:27 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:27.414760+0000 mon.smithi077 (mon.0) 457 : cluster [DBG] osdmap e19: 2 total, 0 up, 2 in 2024-01-13T09:00:28.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:27 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:27.414889+0000 mon.smithi077 (mon.0) 458 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:00:28.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:27 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:27.415284+0000 mon.smithi077 (mon.0) 459 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-13T09:00:28.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:27 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:27.522444+0000 mon.smithi141 (mon.1) 10 : audit [INF] from='client.? 172.21.15.141:0/419560764' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:00:28.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:27 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:27.522844+0000 mon.smithi077 (mon.0) 460 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:00:28.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:27 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:27.531141+0000 mon.smithi077 (mon.0) 461 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:00:28.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:27 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:27.531196+0000 mon.smithi077 (mon.0) 462 : cluster [DBG] osdmap e20: 1 total, 0 up, 1 in 2024-01-13T09:00:28.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:27 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:27.531500+0000 mon.smithi077 (mon.0) 463 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-13T09:00:28.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:27 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:27.726781+0000 mon.smithi077 (mon.0) 464 : audit [INF] from='client.? 172.21.15.77:0/3184650979' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:00:28.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:27 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:27.735165+0000 mon.smithi077 (mon.0) 465 : audit [INF] from='client.? 172.21.15.77:0/3184650979' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:00:28.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:27 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:27.735204+0000 mon.smithi077 (mon.0) 466 : cluster [DBG] osdmap e21: 0 total, 0 up, 0 in 2024-01-13T09:00:28.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:27 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:27.116246+0000 mgr.smithi077.amvfyi (mgr.14180) 116 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:28.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:27 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:27.228250+0000 mon.smithi141 (mon.1) 9 : audit [INF] from='client.? 172.21.15.141:0/278813858' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "276e41e0-b262-4fdc-8ca7-421e3fb7629f"}]: dispatch 2024-01-13T09:00:28.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:27 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:27.228599+0000 mon.smithi077 (mon.0) 451 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "276e41e0-b262-4fdc-8ca7-421e3fb7629f"}]: dispatch 2024-01-13T09:00:28.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:27 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:27.236464+0000 mon.smithi077 (mon.0) 452 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "276e41e0-b262-4fdc-8ca7-421e3fb7629f"}]': finished 2024-01-13T09:00:28.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:27 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:27.236507+0000 mon.smithi077 (mon.0) 453 : cluster [DBG] osdmap e18: 1 total, 0 up, 1 in 2024-01-13T09:00:28.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:27 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:27.236592+0000 mon.smithi077 (mon.0) 454 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:00:28.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:27 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:27.406799+0000 mon.smithi077 (mon.0) 455 : audit [INF] from='client.? 172.21.15.77:0/4108059452' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "435afe84-60df-457c-8c7f-1fed94ed7f02"}]: dispatch 2024-01-13T09:00:28.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:27 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:27.414702+0000 mon.smithi077 (mon.0) 456 : audit [INF] from='client.? 172.21.15.77:0/4108059452' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "435afe84-60df-457c-8c7f-1fed94ed7f02"}]': finished 2024-01-13T09:00:28.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:27 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:27.414760+0000 mon.smithi077 (mon.0) 457 : cluster [DBG] osdmap e19: 2 total, 0 up, 2 in 2024-01-13T09:00:28.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:27 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:27.414889+0000 mon.smithi077 (mon.0) 458 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:00:28.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:27 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:27.415284+0000 mon.smithi077 (mon.0) 459 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-13T09:00:28.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:27 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:27.522444+0000 mon.smithi141 (mon.1) 10 : audit [INF] from='client.? 172.21.15.141:0/419560764' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:00:28.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:27 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:27.522844+0000 mon.smithi077 (mon.0) 460 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:00:28.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:27 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:27.531141+0000 mon.smithi077 (mon.0) 461 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:00:28.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:27 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:27.531196+0000 mon.smithi077 (mon.0) 462 : cluster [DBG] osdmap e20: 1 total, 0 up, 1 in 2024-01-13T09:00:28.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:27 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:27.531500+0000 mon.smithi077 (mon.0) 463 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-13T09:00:28.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:27 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:27.726781+0000 mon.smithi077 (mon.0) 464 : audit [INF] from='client.? 172.21.15.77:0/3184650979' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:00:28.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:27 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:27.735165+0000 mon.smithi077 (mon.0) 465 : audit [INF] from='client.? 172.21.15.77:0/3184650979' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:00:28.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:27 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:27.735204+0000 mon.smithi077 (mon.0) 466 : cluster [DBG] osdmap e21: 0 total, 0 up, 0 in 2024-01-13T09:00:28.797 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:00:29.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T09:00:27.987018+0000 mgr.smithi077.amvfyi (mgr.14180) 117 : cephadm [ERR] executing create_from_spec_one(([('smithi077', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi141', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:00:29.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T09:00:29.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:00:29.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return f(*arg) 2024-01-13T09:00:29.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:00:29.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:00:29.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:00:29.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: code, '\n'.join(err))) 2024-01-13T09:00:29.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T09:00:29.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpa729ry4h:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp_uqol879:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:00:29.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:00:29.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:00:29.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:00:29.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 276e41e0-b262-4fdc-8ca7-421e3fb7629f 2024-01-13T09:00:29.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:00:29.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:00:29.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:00:29.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:00:29.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:00:29.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.prepare() 2024-01-13T09:00:29.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:29.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:29.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:00:29.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:00:29.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:00:29.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:00:29.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:00:29.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:00:29.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:00:29.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T09:00:29.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:00:29.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T09:00:29.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:00:29.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:00:29.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:00:29.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:00:29.079 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:00:29.079 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:00:29.079 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:00:29.079 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:00:29.079 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:00:29.079 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:00:29.079 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T09:00:29.079 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:00:29.079 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:00:29.079 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:00:29.079 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T09:00:29.079 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:29.079 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:29.079 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:00:29.080 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:00:29.080 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:00:29.080 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:00:29.080 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:29.080 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:29.080 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:00:29.080 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:00:29.080 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:00:29.080 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:00:29.080 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:00:29.080 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:00:29.080 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:00:29.081 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:00:29.081 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:29.081 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:29.081 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:00:29.081 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:00:29.081 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:00:29.081 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:00:29.081 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:00:29.082 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T09:00:29.082 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:00:29.082 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: main() 2024-01-13T09:00:29.082 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:00:29.082 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: r = ctx.func(ctx) 2024-01-13T09:00:29.082 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:00:29.082 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:00:29.082 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:00:29.082 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:00:29.082 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:00:29.082 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:00:29.082 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:00:29.082 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:00:29.082 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:00:29.082 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:00:29.083 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:00:29.083 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:00:29.083 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpa729ry4h:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp_uqol879:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:00:29.083 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:28.805346+0000 mon.smithi077 (mon.0) 467 : audit [DBG] from='client.? 172.21.15.77:0/136516019' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:29.227 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:00:29.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T09:00:27.987018+0000 mgr.smithi077.amvfyi (mgr.14180) 117 : cephadm [ERR] executing create_from_spec_one(([('smithi077', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi141', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:00:29.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T09:00:29.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:00:29.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return f(*arg) 2024-01-13T09:00:29.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:00:29.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:00:29.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:00:29.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: code, '\n'.join(err))) 2024-01-13T09:00:29.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T09:00:29.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpa729ry4h:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp_uqol879:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:00:29.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:00:29.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:00:29.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:00:29.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 276e41e0-b262-4fdc-8ca7-421e3fb7629f 2024-01-13T09:00:29.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:00:29.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:00:29.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:00:29.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:00:29.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:00:29.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.prepare() 2024-01-13T09:00:29.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:29.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:29.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:00:29.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:00:29.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:00:29.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:00:29.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:00:29.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:00:29.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:00:29.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T09:00:29.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:00:29.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T09:00:29.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:00:29.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:00:29.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:00:29.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:00:29.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:00:29.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:00:29.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:00:29.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:00:29.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:00:29.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:00:29.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T09:00:29.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:00:29.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:00:29.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:00:29.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T09:00:29.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:29.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:29.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:00:29.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:00:29.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:00:29.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:00:29.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:29.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:29.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:00:29.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:00:29.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:00:29.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:00:29.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:00:29.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:00:29.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:00:29.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:00:29.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:29.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:29.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:00:29.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:00:29.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:00:29.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:00:29.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:00:29.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T09:00:29.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:00:29.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: main() 2024-01-13T09:00:29.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:00:29.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: r = ctx.func(ctx) 2024-01-13T09:00:29.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:00:29.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:00:29.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:00:29.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:00:29.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:00:29.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:00:29.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:00:29.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:00:29.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:00:29.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:00:29.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:00:29.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:00:29.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpa729ry4h:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp_uqol879:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:00:29.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:28.805346+0000 mon.smithi077 (mon.0) 467 : audit [DBG] from='client.? 172.21.15.77:0/136516019' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:30.228 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:00:30.261 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T09:00:28.385864+0000 mgr.smithi077.amvfyi (mgr.14180) 118 : cephadm [ERR] executing create_from_spec_one(([('smithi077', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi141', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:00:30.262 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T09:00:30.262 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:00:30.262 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return f(*arg) 2024-01-13T09:00:30.262 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:00:30.262 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:00:30.262 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:00:30.262 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: code, '\n'.join(err))) 2024-01-13T09:00:30.262 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi077/config 2024-01-13T09:00:30.262 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnwfq6_st:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpqrs8myag:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:00:30.262 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:00:30.262 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:00:30.262 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:00:30.262 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 435afe84-60df-457c-8c7f-1fed94ed7f02 2024-01-13T09:00:30.263 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:00:30.263 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 --yes-i-really-mean-it 2024-01-13T09:00:30.263 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr stderr: purged osd.1 2024-01-13T09:00:30.263 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:00:30.263 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:00:30.263 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.prepare() 2024-01-13T09:00:30.263 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:30.263 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:30.263 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:00:30.263 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:00:30.263 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:00:30.263 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:00:30.263 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:00:30.263 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:00:30.263 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:00:30.264 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T09:00:30.264 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:00:30.264 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T09:00:30.264 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:00:30.264 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:00:30.264 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:00:30.264 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:00:30.264 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:00:30.264 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:00:30.264 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:00:30.265 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:00:30.265 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:00:30.265 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:00:30.265 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T09:00:30.265 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:00:30.265 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:00:30.265 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:00:30.265 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T09:00:30.265 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:30.265 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:30.265 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:00:30.265 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:00:30.265 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:00:30.266 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:00:30.266 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:30.266 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:30.266 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:00:30.266 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:00:30.266 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:00:30.266 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:00:30.266 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:00:30.266 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:00:30.266 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:00:30.266 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:00:30.266 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:30.266 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:30.266 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:00:30.266 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:00:30.267 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:00:30.267 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:00:30.267 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 1 2024-01-13T09:00:30.267 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T09:00:30.267 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:00:30.267 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: main() 2024-01-13T09:00:30.267 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:00:30.267 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: r = ctx.func(ctx) 2024-01-13T09:00:30.267 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:00:30.267 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:00:30.267 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:00:30.267 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:00:30.267 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:00:30.267 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:00:30.267 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:00:30.268 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:00:30.268 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:00:30.268 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:00:30.268 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:00:30.268 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:00:30.268 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnwfq6_st:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpqrs8myag:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:00:30.268 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T09:00:28.387376+0000 mgr.smithi077.amvfyi (mgr.14180) 119 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T09:00:30.268 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: service_id: all-available-devices 2024-01-13T09:00:30.268 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: service_name: osd.all-available-devices 2024-01-13T09:00:30.268 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: placement: 2024-01-13T09:00:30.268 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: host_pattern: '*' 2024-01-13T09:00:30.268 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: spec: 2024-01-13T09:00:30.268 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: data_devices: 2024-01-13T09:00:30.268 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: all: true 2024-01-13T09:00:30.269 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: filter_logic: AND 2024-01-13T09:00:30.269 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: objectstore: bluestore 2024-01-13T09:00:30.269 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T09:00:30.270 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpa729ry4h:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp_uqol879:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:00:30.270 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:00:30.270 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:00:30.270 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:00:30.270 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 276e41e0-b262-4fdc-8ca7-421e3fb7629f 2024-01-13T09:00:30.270 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:00:30.271 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:00:30.271 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:00:30.271 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:00:30.271 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:00:30.271 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.prepare() 2024-01-13T09:00:30.271 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:30.271 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:30.271 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:00:30.271 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:00:30.271 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:00:30.271 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:00:30.271 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:00:30.271 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:00:30.271 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:00:30.271 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T09:00:30.272 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:00:30.272 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T09:00:30.272 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:00:30.272 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:00:30.272 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:00:30.272 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:00:30.272 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:00:30.272 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:00:30.272 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:00:30.272 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:00:30.272 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:00:30.272 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:00:30.272 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T09:00:30.272 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:00:30.272 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:00:30.273 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:00:30.273 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T09:00:30.273 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:30.273 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:30.273 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:00:30.273 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:00:30.273 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:00:30.273 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:00:30.273 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:30.273 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:30.273 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:00:30.273 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:00:30.273 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:00:30.274 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:00:30.274 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:00:30.274 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:00:30.274 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:00:30.274 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:00:30.274 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:30.274 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:30.274 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:00:30.274 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:00:30.274 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:00:30.274 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:00:30.274 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:00:30.274 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T09:00:30.274 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:00:30.274 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: main() 2024-01-13T09:00:30.275 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:00:30.275 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: r = ctx.func(ctx) 2024-01-13T09:00:30.275 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:00:30.275 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:00:30.275 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:00:30.275 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:00:30.275 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:00:30.275 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:00:30.275 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:00:30.275 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:00:30.275 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:00:30.275 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:00:30.275 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:00:30.275 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:00:30.276 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpa729ry4h:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp_uqol879:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:00:30.276 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T09:00:30.276 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T09:00:30.276 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: if self._apply_service(spec): 2024-01-13T09:00:30.276 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T09:00:30.276 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T09:00:30.276 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T09:00:30.276 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T09:00:30.276 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T09:00:30.276 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T09:00:30.276 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T09:00:30.276 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T09:00:30.276 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T09:00:30.276 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: raise self._value 2024-01-13T09:00:30.276 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T09:00:30.277 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: result = (True, func(*args, **kwds)) 2024-01-13T09:00:30.277 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T09:00:30.277 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return list(map(*args)) 2024-01-13T09:00:30.277 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:00:30.277 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return f(*arg) 2024-01-13T09:00:30.277 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:00:30.277 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:00:30.277 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:00:30.277 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: code, '\n'.join(err))) 2024-01-13T09:00:30.277 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T09:00:30.277 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpa729ry4h:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp_uqol879:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:00:30.277 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:00:30.277 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:00:30.277 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:00:30.277 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 276e41e0-b262-4fdc-8ca7-421e3fb7629f 2024-01-13T09:00:30.278 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:00:30.278 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:00:30.278 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:00:30.278 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:00:30.278 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:00:30.278 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.prepare() 2024-01-13T09:00:30.278 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:30.278 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:30.278 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:00:30.278 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:00:30.278 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:00:30.278 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:00:30.278 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:00:30.278 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:00:30.279 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:00:30.279 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T09:00:30.279 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:00:30.279 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T09:00:30.279 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:00:30.279 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:00:30.279 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:00:30.279 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:00:30.279 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:00:30.279 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:00:30.279 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:00:30.279 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:00:30.279 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:00:30.279 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:00:30.279 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T09:00:30.280 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:00:30.280 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:00:30.280 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:00:30.280 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T09:00:30.280 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:30.280 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:30.280 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:00:30.280 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:00:30.280 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:00:30.280 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:00:30.280 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:30.280 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:30.280 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:00:30.280 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:00:30.280 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:00:30.281 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:00:30.281 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:00:30.281 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:00:30.281 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:00:30.281 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:00:30.281 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:30.281 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:30.281 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:00:30.281 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:00:30.281 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:00:30.281 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:00:30.281 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:00:30.281 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T09:00:30.281 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:00:30.282 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: main() 2024-01-13T09:00:30.282 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:00:30.282 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: r = ctx.func(ctx) 2024-01-13T09:00:30.282 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:00:30.282 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:00:30.282 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:00:30.282 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:00:30.282 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:00:30.282 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:00:30.282 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:00:30.282 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:00:30.282 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:00:30.282 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:00:30.282 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:00:30.282 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:00:30.283 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpa729ry4h:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp_uqol879:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:00:30.283 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:28.388004+0000 mgr.smithi077.amvfyi (mgr.14180) 120 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:30.283 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:28.965342+0000 mon.smithi077 (mon.0) 468 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T09:00:30.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T09:00:28.385864+0000 mgr.smithi077.amvfyi (mgr.14180) 118 : cephadm [ERR] executing create_from_spec_one(([('smithi077', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi141', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:00:30.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T09:00:30.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:00:30.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return f(*arg) 2024-01-13T09:00:30.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:00:30.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:00:30.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:00:30.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: code, '\n'.join(err))) 2024-01-13T09:00:30.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi077/config 2024-01-13T09:00:30.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnwfq6_st:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpqrs8myag:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:00:30.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:00:30.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:00:30.381 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:00:30.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 435afe84-60df-457c-8c7f-1fed94ed7f02 2024-01-13T09:00:30.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:00:30.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 --yes-i-really-mean-it 2024-01-13T09:00:30.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr stderr: purged osd.1 2024-01-13T09:00:30.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:00:30.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:00:30.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.prepare() 2024-01-13T09:00:30.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:30.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:30.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:00:30.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:00:30.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:00:30.382 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:00:30.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:00:30.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:00:30.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:00:30.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T09:00:30.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:00:30.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T09:00:30.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:00:30.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:00:30.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:00:30.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:00:30.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:00:30.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:00:30.383 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:00:30.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:00:30.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:00:30.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:00:30.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T09:00:30.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:00:30.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:00:30.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:00:30.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T09:00:30.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:30.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:30.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:00:30.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:00:30.384 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:00:30.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:00:30.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:30.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:30.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:00:30.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:00:30.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:00:30.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:00:30.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:00:30.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:00:30.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:00:30.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:00:30.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:30.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:30.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:00:30.385 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:00:30.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:00:30.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:00:30.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 1 2024-01-13T09:00:30.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T09:00:30.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:00:30.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: main() 2024-01-13T09:00:30.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:00:30.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: r = ctx.func(ctx) 2024-01-13T09:00:30.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:00:30.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:00:30.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:00:30.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:00:30.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:00:30.386 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:00:30.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:00:30.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:00:30.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:00:30.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:00:30.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:00:30.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:00:30.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnwfq6_st:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpqrs8myag:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:00:30.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T09:00:28.387376+0000 mgr.smithi077.amvfyi (mgr.14180) 119 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T09:00:30.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: service_id: all-available-devices 2024-01-13T09:00:30.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: service_name: osd.all-available-devices 2024-01-13T09:00:30.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: placement: 2024-01-13T09:00:30.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: host_pattern: '*' 2024-01-13T09:00:30.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: spec: 2024-01-13T09:00:30.387 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: data_devices: 2024-01-13T09:00:30.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: all: true 2024-01-13T09:00:30.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: filter_logic: AND 2024-01-13T09:00:30.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: objectstore: bluestore 2024-01-13T09:00:30.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T09:00:30.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpa729ry4h:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp_uqol879:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:00:30.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:00:30.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:00:30.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:00:30.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 276e41e0-b262-4fdc-8ca7-421e3fb7629f 2024-01-13T09:00:30.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:00:30.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:00:30.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:00:30.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:00:30.388 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:00:30.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.prepare() 2024-01-13T09:00:30.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:30.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:30.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:00:30.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:00:30.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:00:30.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:00:30.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:00:30.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:00:30.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:00:30.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T09:00:30.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:00:30.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T09:00:30.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:00:30.389 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:00:30.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:00:30.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:00:30.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:00:30.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:00:30.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:00:30.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:00:30.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:00:30.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:00:30.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T09:00:30.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:00:30.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:00:30.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:00:30.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T09:00:30.390 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:30.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:30.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:00:30.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:00:30.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:00:30.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:00:30.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:30.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:30.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:00:30.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:00:30.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:00:30.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:00:30.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:00:30.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:00:30.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:00:30.391 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:00:30.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:30.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:30.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:00:30.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:00:30.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:00:30.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:00:30.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:00:30.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T09:00:30.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:00:30.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: main() 2024-01-13T09:00:30.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:00:30.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: r = ctx.func(ctx) 2024-01-13T09:00:30.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:00:30.392 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:00:30.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:00:30.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:00:30.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:00:30.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:00:30.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:00:30.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:00:30.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:00:30.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:00:30.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:00:30.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:00:30.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpa729ry4h:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp_uqol879:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:00:30.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T09:00:30.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T09:00:30.393 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: if self._apply_service(spec): 2024-01-13T09:00:30.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T09:00:30.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T09:00:30.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T09:00:30.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T09:00:30.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T09:00:30.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T09:00:30.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T09:00:30.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T09:00:30.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T09:00:30.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: raise self._value 2024-01-13T09:00:30.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T09:00:30.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: result = (True, func(*args, **kwds)) 2024-01-13T09:00:30.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T09:00:30.394 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return list(map(*args)) 2024-01-13T09:00:30.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:00:30.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return f(*arg) 2024-01-13T09:00:30.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:00:30.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:00:30.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:00:30.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: code, '\n'.join(err))) 2024-01-13T09:00:30.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T09:00:30.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpa729ry4h:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp_uqol879:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:00:30.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:00:30.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:00:30.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:00:30.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 276e41e0-b262-4fdc-8ca7-421e3fb7629f 2024-01-13T09:00:30.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:00:30.395 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:00:30.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:00:30.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:00:30.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:00:30.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.prepare() 2024-01-13T09:00:30.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:30.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:30.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:00:30.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:00:30.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:00:30.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:00:30.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:00:30.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:00:30.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:00:30.396 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T09:00:30.397 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:00:30.397 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T09:00:30.397 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:00:30.397 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:00:30.397 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:00:30.398 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:00:30.398 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:00:30.398 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:00:30.398 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:00:30.398 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:00:30.398 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:00:30.398 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:00:30.398 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T09:00:30.398 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:00:30.398 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:00:30.398 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:00:30.398 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T09:00:30.398 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:30.398 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:30.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:00:30.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:00:30.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:00:30.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:00:30.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:30.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:30.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:00:30.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:00:30.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:00:30.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:00:30.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:00:30.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:00:30.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:00:30.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:00:30.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:00:30.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:00:30.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:00:30.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:00:30.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:00:30.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:00:30.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:00:30.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T09:00:30.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:00:30.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: main() 2024-01-13T09:00:30.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:00:30.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: r = ctx.func(ctx) 2024-01-13T09:00:30.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:00:30.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:00:30.401 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:00:30.401 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:00:30.401 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:00:30.401 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:00:30.401 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:00:30.401 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:00:30.401 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:00:30.401 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:00:30.401 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:00:30.401 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:00:30.401 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpa729ry4h:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp_uqol879:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:00:30.401 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:28.388004+0000 mgr.smithi077.amvfyi (mgr.14180) 120 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:30.401 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:28.965342+0000 mon.smithi077 (mon.0) 468 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T09:00:31.657 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:00:32.034 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:00:32.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:31 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:30.388399+0000 mgr.smithi077.amvfyi (mgr.14180) 121 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:32.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:31 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:31.664274+0000 mon.smithi077 (mon.0) 469 : audit [DBG] from='client.? 172.21.15.77:0/3208086457' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:32.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:31 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:30.388399+0000 mgr.smithi077.amvfyi (mgr.14180) 121 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:32.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:31 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:31.664274+0000 mon.smithi077 (mon.0) 469 : audit [DBG] from='client.? 172.21.15.77:0/3208086457' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:33.035 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:00:34.307 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:00:34.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:32.388973+0000 mgr.smithi077.amvfyi (mgr.14180) 122 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:34.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:32.388973+0000 mgr.smithi077.amvfyi (mgr.14180) 122 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:34.666 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:00:35.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:34.314640+0000 mon.smithi077 (mon.0) 470 : audit [DBG] from='client.? 172.21.15.77:0/70383573' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:35.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:34.314640+0000 mon.smithi077 (mon.0) 470 : audit [DBG] from='client.? 172.21.15.77:0/70383573' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:35.667 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:00:36.320 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:35 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:34.389512+0000 mgr.smithi077.amvfyi (mgr.14180) 123 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:36.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:35 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:34.389512+0000 mgr.smithi077.amvfyi (mgr.14180) 123 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:37.040 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:00:37.392 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:00:38.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:36.389931+0000 mgr.smithi077.amvfyi (mgr.14180) 124 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:38.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:37.047390+0000 mon.smithi077 (mon.0) 471 : audit [DBG] from='client.? 172.21.15.77:0/2267113501' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:38.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:36.389931+0000 mgr.smithi077.amvfyi (mgr.14180) 124 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:38.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:37.047390+0000 mon.smithi077 (mon.0) 471 : audit [DBG] from='client.? 172.21.15.77:0/2267113501' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:38.393 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:00:39.823 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:00:40.156 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:39 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:38.390670+0000 mgr.smithi077.amvfyi (mgr.14180) 125 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:40.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:39 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:39.830985+0000 mon.smithi077 (mon.0) 472 : audit [DBG] from='client.? 172.21.15.77:0/2067097393' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:40.328 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:00:40.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:39 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:38.390670+0000 mgr.smithi077.amvfyi (mgr.14180) 125 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:40.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:39 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:39.830985+0000 mon.smithi077 (mon.0) 472 : audit [DBG] from='client.? 172.21.15.77:0/2067097393' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:41.330 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:00:42.304 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:42 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:40.391075+0000 mgr.smithi077.amvfyi (mgr.14180) 126 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:42.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:42 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:40.391075+0000 mgr.smithi077.amvfyi (mgr.14180) 126 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:42.678 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:00:43.092 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:00:43.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:43 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:42.686567+0000 mon.smithi077 (mon.0) 473 : audit [DBG] from='client.? 172.21.15.77:0/4250864202' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:43.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:43 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:42.686567+0000 mon.smithi077 (mon.0) 473 : audit [DBG] from='client.? 172.21.15.77:0/4250864202' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:44.093 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:00:44.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:44 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:42.391806+0000 mgr.smithi077.amvfyi (mgr.14180) 127 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:44.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:44 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:42.391806+0000 mgr.smithi077.amvfyi (mgr.14180) 127 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:45.420 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:00:45.782 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:00:46.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:44.392200+0000 mgr.smithi077.amvfyi (mgr.14180) 128 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:46.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:45.428041+0000 mon.smithi077 (mon.0) 474 : audit [DBG] from='client.? 172.21.15.77:0/4058828003' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:46.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:46 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:44.392200+0000 mgr.smithi077.amvfyi (mgr.14180) 128 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:46.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:46 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:45.428041+0000 mon.smithi077 (mon.0) 474 : audit [DBG] from='client.? 172.21.15.77:0/4058828003' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:46.784 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:00:48.088 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:00:48.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:48 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:46.392610+0000 mgr.smithi077.amvfyi (mgr.14180) 129 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:48.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:48 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:46.392610+0000 mgr.smithi077.amvfyi (mgr.14180) 129 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:48.465 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:00:49.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:49 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:48.095770+0000 mon.smithi077 (mon.0) 475 : audit [DBG] from='client.? 172.21.15.77:0/4252727121' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:49.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:49 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:48.095770+0000 mon.smithi077 (mon.0) 475 : audit [DBG] from='client.? 172.21.15.77:0/4252727121' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:49.467 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:00:50.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:50 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:48.393131+0000 mgr.smithi077.amvfyi (mgr.14180) 130 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:50.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:50 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:48.393131+0000 mgr.smithi077.amvfyi (mgr.14180) 130 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:50.828 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:00:51.155 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:51 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:50.836242+0000 mon.smithi077 (mon.0) 476 : audit [DBG] from='client.? 172.21.15.77:0/4001637272' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:51.215 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:00:51.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:51 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:50.836242+0000 mon.smithi077 (mon.0) 476 : audit [DBG] from='client.? 172.21.15.77:0/4001637272' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:52.216 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:00:52.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:50.393625+0000 mgr.smithi077.amvfyi (mgr.14180) 131 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:52.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:50.393625+0000 mgr.smithi077.amvfyi (mgr.14180) 131 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:53.598 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:00:53.964 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:00:54.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:54 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:52.394034+0000 mgr.smithi077.amvfyi (mgr.14180) 132 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:54.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:54 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:53.605702+0000 mon.smithi077 (mon.0) 477 : audit [DBG] from='client.? 172.21.15.77:0/795381787' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:54.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:54 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:52.394034+0000 mgr.smithi077.amvfyi (mgr.14180) 132 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:54.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:54 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:53.605702+0000 mon.smithi077 (mon.0) 477 : audit [DBG] from='client.? 172.21.15.77:0/795381787' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:54.965 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:00:56.259 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:00:56.285 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:56 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:54.394542+0000 mgr.smithi077.amvfyi (mgr.14180) 133 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:56.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:56 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:54.394542+0000 mgr.smithi077.amvfyi (mgr.14180) 133 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:56.745 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:00:57.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:57 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:56.267466+0000 mon.smithi077 (mon.0) 478 : audit [DBG] from='client.? 172.21.15.77:0/2421421216' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:57.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:57 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:56.267466+0000 mon.smithi077 (mon.0) 478 : audit [DBG] from='client.? 172.21.15.77:0/2421421216' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:00:57.746 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:00:58.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:00:58 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:56.394962+0000 mgr.smithi077.amvfyi (mgr.14180) 134 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:58.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:00:58 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:56.394962+0000 mgr.smithi077.amvfyi (mgr.14180) 134 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:00:59.098 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:00:59.472 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:01:00.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:00:58.395440+0000 mgr.smithi077.amvfyi (mgr.14180) 135 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:00.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:00:59.105788+0000 mon.smithi077 (mon.0) 479 : audit [DBG] from='client.? 172.21.15.77:0/1794498245' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:00.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:00:58.395440+0000 mgr.smithi077.amvfyi (mgr.14180) 135 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:00.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:00:59.105788+0000 mon.smithi077 (mon.0) 479 : audit [DBG] from='client.? 172.21.15.77:0/1794498245' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:00.473 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:01:01.727 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:01:02.097 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:01:02.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:02 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:01:00.395908+0000 mgr.smithi077.amvfyi (mgr.14180) 136 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:02.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:02 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:01.733959+0000 mon.smithi077 (mon.0) 480 : audit [DBG] from='client.? 172.21.15.77:0/342352237' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:02.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:02 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:01:00.395908+0000 mgr.smithi077.amvfyi (mgr.14180) 136 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:02.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:02 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:01.733959+0000 mon.smithi077 (mon.0) 480 : audit [DBG] from='client.? 172.21.15.77:0/342352237' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:03.099 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:01:04.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:04 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:01:02.396589+0000 mgr.smithi077.amvfyi (mgr.14180) 137 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:04.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:04 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:01:02.396589+0000 mgr.smithi077.amvfyi (mgr.14180) 137 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:04.383 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:01:04.763 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:01:05.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:05 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:04.390539+0000 mon.smithi077 (mon.0) 481 : audit [DBG] from='client.? 172.21.15.77:0/663866559' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:05.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:05 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:04.390539+0000 mon.smithi077 (mon.0) 481 : audit [DBG] from='client.? 172.21.15.77:0/663866559' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:05.764 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:01:06.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:06 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:01:04.396952+0000 mgr.smithi077.amvfyi (mgr.14180) 138 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:06.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:06 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:01:04.396952+0000 mgr.smithi077.amvfyi (mgr.14180) 138 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:07.136 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:01:07.530 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:01:08.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:08 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:01:06.397444+0000 mgr.smithi077.amvfyi (mgr.14180) 139 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:08.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:08 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:07.144700+0000 mon.smithi077 (mon.0) 482 : audit [DBG] from='client.? 172.21.15.77:0/3384859545' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:08.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:08 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:01:06.397444+0000 mgr.smithi077.amvfyi (mgr.14180) 139 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:08.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:08 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:07.144700+0000 mon.smithi077 (mon.0) 482 : audit [DBG] from='client.? 172.21.15.77:0/3384859545' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:08.531 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:01:09.887 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:01:10.219 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:10 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:01:08.397937+0000 mgr.smithi077.amvfyi (mgr.14180) 140 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:10.220 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:10 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:09.894436+0000 mon.smithi077 (mon.0) 483 : audit [DBG] from='client.? 172.21.15.77:0/1074417804' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:10.262 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:01:10.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:10 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:01:08.397937+0000 mgr.smithi077.amvfyi (mgr.14180) 140 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:10.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:10 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:09.894436+0000 mon.smithi077 (mon.0) 483 : audit [DBG] from='client.? 172.21.15.77:0/1074417804' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:11.264 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:01:12.249 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:12 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:01:10.398448+0000 mgr.smithi077.amvfyi (mgr.14180) 141 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:12.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:12 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:01:10.398448+0000 mgr.smithi077.amvfyi (mgr.14180) 141 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:12.590 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:01:12.971 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:01:13.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:13 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:12.597781+0000 mon.smithi077 (mon.0) 484 : audit [DBG] from='client.? 172.21.15.77:0/1038420603' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:13.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:13 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:12.597781+0000 mon.smithi077 (mon.0) 484 : audit [DBG] from='client.? 172.21.15.77:0/1038420603' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:13.972 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:01:14.315 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:01:12.398809+0000 mgr.smithi077.amvfyi (mgr.14180) 142 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:14.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:01:12.398809+0000 mgr.smithi077.amvfyi (mgr.14180) 142 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:15.284 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:01:15.680 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:01:16.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:16 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:01:14.399381+0000 mgr.smithi077.amvfyi (mgr.14180) 143 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:16.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:16 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:15.291626+0000 mon.smithi077 (mon.0) 485 : audit [DBG] from='client.? 172.21.15.77:0/2203907837' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:16.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:16 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:01:14.399381+0000 mgr.smithi077.amvfyi (mgr.14180) 143 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:16.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:16 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:15.291626+0000 mon.smithi077 (mon.0) 485 : audit [DBG] from='client.? 172.21.15.77:0/2203907837' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:16.681 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:01:17.997 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:01:18.323 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:18 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:01:16.400025+0000 mgr.smithi077.amvfyi (mgr.14180) 144 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:18.323 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:18 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:18.005345+0000 mon.smithi077 (mon.0) 486 : audit [DBG] from='client.? 172.21.15.77:0/4084288087' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:18.379 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:01:18.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:18 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:01:16.400025+0000 mgr.smithi077.amvfyi (mgr.14180) 144 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:18.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:18 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:18.005345+0000 mon.smithi077 (mon.0) 486 : audit [DBG] from='client.? 172.21.15.77:0/4084288087' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:19.380 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:01:20.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:20 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:01:18.400698+0000 mgr.smithi077.amvfyi (mgr.14180) 145 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:20.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:20 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:01:18.400698+0000 mgr.smithi077.amvfyi (mgr.14180) 145 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:20.675 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:01:21.036 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:01:21.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:20.683110+0000 mon.smithi077 (mon.0) 487 : audit [DBG] from='client.? 172.21.15.77:0/1445561448' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:21.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:20.683110+0000 mon.smithi077 (mon.0) 487 : audit [DBG] from='client.? 172.21.15.77:0/1445561448' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:22.037 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:01:22.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:01:20.401047+0000 mgr.smithi077.amvfyi (mgr.14180) 146 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:22.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:01:20.401047+0000 mgr.smithi077.amvfyi (mgr.14180) 146 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:23.417 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:01:23.819 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:01:24.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:24 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:01:22.401576+0000 mgr.smithi077.amvfyi (mgr.14180) 147 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:24.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:24 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:23.425277+0000 mon.smithi077 (mon.0) 488 : audit [DBG] from='client.? 172.21.15.77:0/1678668646' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:24.575 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:24 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:01:22.401576+0000 mgr.smithi077.amvfyi (mgr.14180) 147 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:24.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:24 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:23.425277+0000 mon.smithi077 (mon.0) 488 : audit [DBG] from='client.? 172.21.15.77:0/1678668646' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:24.820 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:01:26.133 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:01:26.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:26 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:01:24.402168+0000 mgr.smithi077.amvfyi (mgr.14180) 148 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:26.442 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:26 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:01:24.402168+0000 mgr.smithi077.amvfyi (mgr.14180) 148 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:26.518 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:01:27.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:27 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:26.140781+0000 mon.smithi077 (mon.0) 489 : audit [DBG] from='client.? 172.21.15.77:0/1464289312' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:27.519 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:01:27.551 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:27 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:26.140781+0000 mon.smithi077 (mon.0) 489 : audit [DBG] from='client.? 172.21.15.77:0/1464289312' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:28.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:01:26.402664+0000 mgr.smithi077.amvfyi (mgr.14180) 149 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:28.472 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:01:26.402664+0000 mgr.smithi077.amvfyi (mgr.14180) 149 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:28.828 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:01:29.206 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:01:29.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:28.389224+0000 mon.smithi077 (mon.0) 490 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T09:01:29.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:28.739956+0000 mon.smithi077 (mon.0) 491 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:01:29.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:28.740584+0000 mon.smithi077 (mon.0) 492 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T09:01:29.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:28.744779+0000 mon.smithi077 (mon.0) 493 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T09:01:29.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:28.748221+0000 mon.smithi077 (mon.0) 494 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T09:01:29.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:28.750783+0000 mon.smithi077 (mon.0) 495 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:01:29.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:28.751794+0000 mon.smithi077 (mon.0) 496 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:01:29.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:28.751954+0000 mon.smithi077 (mon.0) 497 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:01:29.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:28.752392+0000 mon.smithi077 (mon.0) 498 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:01:29.380 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:28.836012+0000 mon.smithi077 (mon.0) 499 : audit [DBG] from='client.? 172.21.15.77:0/963496081' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:29.402 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:28.389224+0000 mon.smithi077 (mon.0) 490 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T09:01:29.402 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:28.739956+0000 mon.smithi077 (mon.0) 491 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:01:29.402 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:28.740584+0000 mon.smithi077 (mon.0) 492 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T09:01:29.402 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:28.744779+0000 mon.smithi077 (mon.0) 493 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T09:01:29.402 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:28.748221+0000 mon.smithi077 (mon.0) 494 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T09:01:29.402 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:28.750783+0000 mon.smithi077 (mon.0) 495 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:01:29.402 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:28.751794+0000 mon.smithi077 (mon.0) 496 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:01:29.402 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:28.751954+0000 mon.smithi077 (mon.0) 497 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:01:29.403 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:28.752392+0000 mon.smithi077 (mon.0) 498 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:01:29.403 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:28.836012+0000 mon.smithi077 (mon.0) 499 : audit [DBG] from='client.? 172.21.15.77:0/963496081' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:30.207 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:01:30.513 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:30 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:01:28.403030+0000 mgr.smithi077.amvfyi (mgr.14180) 150 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:30.514 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:30 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:01:28.745114+0000 mgr.smithi077.amvfyi (mgr.14180) 151 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:30.514 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:30 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:01:29.134023+0000 mon.smithi077 (mon.0) 500 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T09:01:30.514 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:30 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:01:29.134098+0000 mon.smithi077 (mon.0) 501 : cluster [INF] Cluster is now healthy 2024-01-13T09:01:30.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:30 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:01:28.403030+0000 mgr.smithi077.amvfyi (mgr.14180) 150 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:30.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:30 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:01:28.745114+0000 mgr.smithi077.amvfyi (mgr.14180) 151 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:30.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:30 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:01:29.134023+0000 mon.smithi077 (mon.0) 500 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T09:01:30.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:30 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:01:29.134098+0000 mon.smithi077 (mon.0) 501 : cluster [INF] Cluster is now healthy 2024-01-13T09:01:31.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:31 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:30.757271+0000 mon.smithi141 (mon.1) 11 : audit [INF] from='client.? 172.21.15.141:0/2462603897' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91ed9908-ebe2-414d-9bf2-3b644218d7c2"}]: dispatch 2024-01-13T09:01:31.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:31 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:30.757595+0000 mon.smithi077 (mon.0) 502 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91ed9908-ebe2-414d-9bf2-3b644218d7c2"}]: dispatch 2024-01-13T09:01:31.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:31 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:30.761177+0000 mon.smithi077 (mon.0) 503 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91ed9908-ebe2-414d-9bf2-3b644218d7c2"}]': finished 2024-01-13T09:01:31.399 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:31 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:01:30.761251+0000 mon.smithi077 (mon.0) 504 : cluster [DBG] osdmap e22: 1 total, 0 up, 1 in 2024-01-13T09:01:31.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:31 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:30.761350+0000 mon.smithi077 (mon.0) 505 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:01:31.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:31 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:31.050236+0000 mon.smithi141 (mon.1) 12 : audit [INF] from='client.? 172.21.15.141:0/1134294245' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:01:31.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:31 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:31.050563+0000 mon.smithi077 (mon.0) 506 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:01:31.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:31 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:31.058755+0000 mon.smithi077 (mon.0) 507 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:01:31.400 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:31 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:01:31.058810+0000 mon.smithi077 (mon.0) 508 : cluster [DBG] osdmap e23: 0 total, 0 up, 0 in 2024-01-13T09:01:31.508 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:31 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:30.757271+0000 mon.smithi141 (mon.1) 11 : audit [INF] from='client.? 172.21.15.141:0/2462603897' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91ed9908-ebe2-414d-9bf2-3b644218d7c2"}]: dispatch 2024-01-13T09:01:31.508 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:31 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:30.757595+0000 mon.smithi077 (mon.0) 502 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91ed9908-ebe2-414d-9bf2-3b644218d7c2"}]: dispatch 2024-01-13T09:01:31.508 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:31 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:30.761177+0000 mon.smithi077 (mon.0) 503 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91ed9908-ebe2-414d-9bf2-3b644218d7c2"}]': finished 2024-01-13T09:01:31.508 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:31 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:01:30.761251+0000 mon.smithi077 (mon.0) 504 : cluster [DBG] osdmap e22: 1 total, 0 up, 1 in 2024-01-13T09:01:31.508 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:31 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:30.761350+0000 mon.smithi077 (mon.0) 505 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:01:31.508 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:31 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:31.050236+0000 mon.smithi141 (mon.1) 12 : audit [INF] from='client.? 172.21.15.141:0/1134294245' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:01:31.508 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:31 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:31.050563+0000 mon.smithi077 (mon.0) 506 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:01:31.509 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:31 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:31.058755+0000 mon.smithi077 (mon.0) 507 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:01:31.509 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:31 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:01:31.058810+0000 mon.smithi077 (mon.0) 508 : cluster [DBG] osdmap e23: 0 total, 0 up, 0 in 2024-01-13T09:01:31.524 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:01:31.893 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:01:32.575 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:32 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:01:30.745351+0000 mgr.smithi077.amvfyi (mgr.14180) 152 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:32.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:32 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:31.143942+0000 mon.smithi077 (mon.0) 509 : audit [INF] from='client.? 172.21.15.77:0/1546232186' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5592398e-d910-4474-904a-59c4a8cd3816"}]: dispatch 2024-01-13T09:01:32.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:32 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:31.154431+0000 mon.smithi077 (mon.0) 510 : audit [INF] from='client.? 172.21.15.77:0/1546232186' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5592398e-d910-4474-904a-59c4a8cd3816"}]': finished 2024-01-13T09:01:32.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:32 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:01:31.154469+0000 mon.smithi077 (mon.0) 511 : cluster [DBG] osdmap e24: 1 total, 0 up, 1 in 2024-01-13T09:01:32.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:32 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:31.154576+0000 mon.smithi077 (mon.0) 512 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:01:32.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:32 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:31.465378+0000 mon.smithi077 (mon.0) 513 : audit [INF] from='client.? 172.21.15.77:0/82355921' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:01:32.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:32 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:31.469671+0000 mon.smithi077 (mon.0) 514 : audit [INF] from='client.? 172.21.15.77:0/82355921' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:01:32.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:32 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:01:31.469709+0000 mon.smithi077 (mon.0) 515 : cluster [DBG] osdmap e25: 0 total, 0 up, 0 in 2024-01-13T09:01:32.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:32 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:31.532430+0000 mon.smithi077 (mon.0) 516 : audit [DBG] from='client.? 172.21.15.77:0/1247692317' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:32.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:01:30.745351+0000 mgr.smithi077.amvfyi (mgr.14180) 152 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:32.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:31.143942+0000 mon.smithi077 (mon.0) 509 : audit [INF] from='client.? 172.21.15.77:0/1546232186' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5592398e-d910-4474-904a-59c4a8cd3816"}]: dispatch 2024-01-13T09:01:32.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:31.154431+0000 mon.smithi077 (mon.0) 510 : audit [INF] from='client.? 172.21.15.77:0/1546232186' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5592398e-d910-4474-904a-59c4a8cd3816"}]': finished 2024-01-13T09:01:32.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:01:31.154469+0000 mon.smithi077 (mon.0) 511 : cluster [DBG] osdmap e24: 1 total, 0 up, 1 in 2024-01-13T09:01:32.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:31.154576+0000 mon.smithi077 (mon.0) 512 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:01:32.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:31.465378+0000 mon.smithi077 (mon.0) 513 : audit [INF] from='client.? 172.21.15.77:0/82355921' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:01:32.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:31.469671+0000 mon.smithi077 (mon.0) 514 : audit [INF] from='client.? 172.21.15.77:0/82355921' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:01:32.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:01:31.469709+0000 mon.smithi077 (mon.0) 515 : cluster [DBG] osdmap e25: 0 total, 0 up, 0 in 2024-01-13T09:01:32.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:31.532430+0000 mon.smithi077 (mon.0) 516 : audit [DBG] from='client.? 172.21.15.77:0/1247692317' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:32.894 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:01:33.229 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T09:01:31.515040+0000 mgr.smithi077.amvfyi (mgr.14180) 153 : cephadm [ERR] executing create_from_spec_one(([('smithi077', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi141', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:01:33.229 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T09:01:33.229 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:01:33.230 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return f(*arg) 2024-01-13T09:01:33.230 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:01:33.230 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:01:33.230 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:01:33.230 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: code, '\n'.join(err))) 2024-01-13T09:01:33.230 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T09:01:33.230 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvldeixcm:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpkhpl3aw6:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:01:33.230 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:01:33.230 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:01:33.230 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:01:33.230 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 91ed9908-ebe2-414d-9bf2-3b644218d7c2 2024-01-13T09:01:33.230 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:01:33.230 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:01:33.230 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:01:33.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:01:33.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:01:33.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.prepare() 2024-01-13T09:01:33.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:01:33.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:01:33.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:01:33.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:01:33.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:01:33.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:01:33.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:01:33.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:01:33.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:01:33.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T09:01:33.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:01:33.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T09:01:33.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:01:33.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:01:33.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:01:33.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:01:33.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:01:33.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:01:33.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:01:33.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:01:33.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:01:33.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:01:33.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T09:01:33.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:01:33.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:01:33.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:01:33.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T09:01:33.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:01:33.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:01:33.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:01:33.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:01:33.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:01:33.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:01:33.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:01:33.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:01:33.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:01:33.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:01:33.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:01:33.233 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:01:33.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:01:33.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:01:33.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:01:33.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:01:33.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:01:33.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:01:33.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:01:33.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:01:33.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:01:33.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:01:33.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:01:33.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T09:01:33.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:01:33.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: main() 2024-01-13T09:01:33.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:01:33.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: r = ctx.func(ctx) 2024-01-13T09:01:33.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:01:33.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:01:33.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:01:33.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:01:33.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:01:33.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:01:33.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:01:33.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:01:33.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:01:33.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:01:33.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:01:33.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:01:33.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvldeixcm:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpkhpl3aw6:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:01:33.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T09:01:31.515040+0000 mgr.smithi077.amvfyi (mgr.14180) 153 : cephadm [ERR] executing create_from_spec_one(([('smithi077', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi141', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:01:33.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T09:01:33.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:01:33.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return f(*arg) 2024-01-13T09:01:33.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:01:33.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:01:33.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:01:33.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: code, '\n'.join(err))) 2024-01-13T09:01:33.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T09:01:33.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvldeixcm:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpkhpl3aw6:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:01:33.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:01:33.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:01:33.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:01:33.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 91ed9908-ebe2-414d-9bf2-3b644218d7c2 2024-01-13T09:01:33.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:01:33.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:01:33.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:01:33.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:01:33.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:01:33.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.prepare() 2024-01-13T09:01:33.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:01:33.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:01:33.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:01:33.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:01:33.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:01:33.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:01:33.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:01:33.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:01:33.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:01:33.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T09:01:33.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:01:33.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T09:01:33.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:01:33.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:01:33.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:01:33.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:01:33.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:01:33.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:01:33.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:01:33.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:01:33.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:01:33.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:01:33.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T09:01:33.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:01:33.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:01:33.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:01:33.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T09:01:33.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:01:33.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:01:33.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:01:33.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:01:33.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:01:33.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:01:33.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:01:33.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:01:33.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:01:33.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:01:33.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:01:33.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:01:33.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:01:33.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:01:33.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:01:33.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:01:33.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:01:33.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:01:33.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:01:33.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:01:33.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:01:33.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:01:33.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:01:33.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T09:01:33.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:01:33.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: main() 2024-01-13T09:01:33.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:01:33.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: r = ctx.func(ctx) 2024-01-13T09:01:33.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:01:33.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:01:33.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:01:33.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:01:33.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:01:33.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:01:33.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:01:33.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:01:33.636 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:01:33.636 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:01:33.636 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:01:33.636 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:01:33.636 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvldeixcm:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpkhpl3aw6:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:01:34.209 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:01:34.511 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T09:01:32.276038+0000 mgr.smithi077.amvfyi (mgr.14180) 154 : cephadm [ERR] executing create_from_spec_one(([('smithi077', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi141', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:01:34.511 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T09:01:34.511 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:01:34.511 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return f(*arg) 2024-01-13T09:01:34.511 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:01:34.511 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:01:34.511 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:01:34.511 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: code, '\n'.join(err))) 2024-01-13T09:01:34.511 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi077/config 2024-01-13T09:01:34.512 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4xu2_iuk:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpvn4_lqsj:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:01:34.512 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:01:34.512 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:01:34.512 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:01:34.512 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 5592398e-d910-4474-904a-59c4a8cd3816 2024-01-13T09:01:34.512 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:01:34.512 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:01:34.512 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:01:34.512 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:01:34.512 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:01:34.512 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.prepare() 2024-01-13T09:01:34.513 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:01:34.513 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:01:34.513 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:01:34.513 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:01:34.513 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:01:34.513 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:01:34.513 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:01:34.513 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:01:34.513 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:01:34.513 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T09:01:34.513 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:01:34.514 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T09:01:34.514 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:01:34.514 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:01:34.514 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:01:34.514 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:01:34.514 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:01:34.514 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:01:34.514 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:01:34.514 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:01:34.514 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:01:34.515 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:01:34.515 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T09:01:34.515 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:01:34.515 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:01:34.515 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:01:34.515 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T09:01:34.515 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:01:34.515 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:01:34.515 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:01:34.515 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:01:34.515 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:01:34.515 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:01:34.516 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:01:34.516 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:01:34.516 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:01:34.516 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:01:34.516 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:01:34.516 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:01:34.516 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:01:34.516 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:01:34.516 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:01:34.516 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:01:34.516 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:01:34.516 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:01:34.517 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:01:34.517 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:01:34.517 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:01:34.517 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:01:34.517 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:01:34.517 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T09:01:34.517 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:01:34.517 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: main() 2024-01-13T09:01:34.517 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:01:34.517 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: r = ctx.func(ctx) 2024-01-13T09:01:34.517 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:01:34.517 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:01:34.518 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:01:34.518 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:01:34.518 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:01:34.518 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:01:34.518 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:01:34.518 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:01:34.518 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:01:34.518 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:01:34.518 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:01:34.518 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:01:34.518 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4xu2_iuk:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpvn4_lqsj:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:01:34.518 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T09:01:32.277470+0000 mgr.smithi077.amvfyi (mgr.14180) 155 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T09:01:34.518 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: service_id: all-available-devices 2024-01-13T09:01:34.519 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: service_name: osd.all-available-devices 2024-01-13T09:01:34.519 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: placement: 2024-01-13T09:01:34.519 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: host_pattern: '*' 2024-01-13T09:01:34.519 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: spec: 2024-01-13T09:01:34.519 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: data_devices: 2024-01-13T09:01:34.519 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: all: true 2024-01-13T09:01:34.519 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: filter_logic: AND 2024-01-13T09:01:34.519 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: objectstore: bluestore 2024-01-13T09:01:34.519 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T09:01:34.519 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvldeixcm:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpkhpl3aw6:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:01:34.519 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:01:34.520 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:01:34.520 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:01:34.520 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 91ed9908-ebe2-414d-9bf2-3b644218d7c2 2024-01-13T09:01:34.520 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:01:34.520 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:01:34.520 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:01:34.520 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:01:34.520 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:01:34.520 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.prepare() 2024-01-13T09:01:34.520 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:01:34.520 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:01:34.520 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:01:34.521 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:01:34.521 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:01:34.521 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:01:34.521 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:01:34.521 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:01:34.521 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:01:34.521 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T09:01:34.521 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:01:34.521 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T09:01:34.521 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:01:34.521 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:01:34.522 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:01:34.522 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:01:34.522 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:01:34.522 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:01:34.522 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:01:34.522 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:01:34.522 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:01:34.522 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:01:34.522 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T09:01:34.522 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:01:34.522 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:01:34.522 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:01:34.523 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T09:01:34.523 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:01:34.523 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:01:34.523 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:01:34.523 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:01:34.523 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:01:34.523 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:01:34.523 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:01:34.523 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:01:34.523 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:01:34.523 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:01:34.523 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:01:34.523 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:01:34.524 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:01:34.524 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:01:34.524 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:01:34.524 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:01:34.524 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:01:34.524 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:01:34.524 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:01:34.524 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:01:34.525 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:01:34.525 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:01:34.525 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:01:34.525 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T09:01:34.525 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:01:34.525 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: main() 2024-01-13T09:01:34.525 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:01:34.525 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: r = ctx.func(ctx) 2024-01-13T09:01:34.525 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:01:34.525 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:01:34.525 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:01:34.525 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:01:34.526 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:01:34.526 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:01:34.526 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:01:34.526 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:01:34.526 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:01:34.526 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:01:34.526 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:01:34.526 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:01:34.526 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvldeixcm:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpkhpl3aw6:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:01:34.526 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T09:01:34.526 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T09:01:34.526 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: if self._apply_service(spec): 2024-01-13T09:01:34.527 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T09:01:34.527 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T09:01:34.527 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T09:01:34.527 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T09:01:34.527 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T09:01:34.527 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T09:01:34.527 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T09:01:34.527 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T09:01:34.527 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T09:01:34.527 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: raise self._value 2024-01-13T09:01:34.527 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T09:01:34.527 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: result = (True, func(*args, **kwds)) 2024-01-13T09:01:34.528 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T09:01:34.528 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return list(map(*args)) 2024-01-13T09:01:34.528 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:01:34.528 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return f(*arg) 2024-01-13T09:01:34.528 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:01:34.528 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:01:34.528 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:01:34.528 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: code, '\n'.join(err))) 2024-01-13T09:01:34.528 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T09:01:34.528 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvldeixcm:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpkhpl3aw6:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:01:34.528 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:01:34.528 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:01:34.529 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:01:34.529 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 91ed9908-ebe2-414d-9bf2-3b644218d7c2 2024-01-13T09:01:34.529 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:01:34.529 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:01:34.529 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:01:34.529 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:01:34.529 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:01:34.530 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.prepare() 2024-01-13T09:01:34.530 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:01:34.530 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:01:34.530 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:01:34.530 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:01:34.530 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:01:34.530 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:01:34.530 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:01:34.530 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:01:34.530 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:01:34.530 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T09:01:34.530 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:01:34.530 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T09:01:34.531 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:01:34.531 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:01:34.531 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:01:34.531 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:01:34.531 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:01:34.531 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:01:34.531 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:01:34.531 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:01:34.531 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:01:34.531 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:01:34.531 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T09:01:34.531 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:01:34.531 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:01:34.532 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:01:34.532 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T09:01:34.532 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:01:34.532 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:01:34.532 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:01:34.532 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:01:34.532 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:01:34.532 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:01:34.532 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:01:34.532 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:01:34.532 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:01:34.533 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:01:34.533 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:01:34.533 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:01:34.533 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:01:34.533 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:01:34.533 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:01:34.533 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:01:34.533 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:01:34.533 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:01:34.533 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:01:34.533 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:01:34.533 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:01:34.534 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:01:34.534 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:01:34.534 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T09:01:34.534 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:01:34.534 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: main() 2024-01-13T09:01:34.534 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:01:34.534 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: r = ctx.func(ctx) 2024-01-13T09:01:34.534 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:01:34.534 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:01:34.534 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:01:34.535 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:01:34.535 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:01:34.535 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:01:34.535 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:01:34.535 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:01:34.535 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:01:34.535 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:01:34.535 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:01:34.535 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:01:34.535 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvldeixcm:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpkhpl3aw6:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:01:34.535 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:01:32.278088+0000 mgr.smithi077.amvfyi (mgr.14180) 156 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:34.535 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:01:33.154314+0000 mon.smithi077 (mon.0) 517 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T09:01:34.592 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:01:34.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T09:01:32.276038+0000 mgr.smithi077.amvfyi (mgr.14180) 154 : cephadm [ERR] executing create_from_spec_one(([('smithi077', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi141', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:01:34.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T09:01:34.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:01:34.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return f(*arg) 2024-01-13T09:01:34.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:01:34.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:01:34.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:01:34.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: code, '\n'.join(err))) 2024-01-13T09:01:34.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi077/config 2024-01-13T09:01:34.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4xu2_iuk:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpvn4_lqsj:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:01:34.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:01:34.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:01:34.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:01:34.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 5592398e-d910-4474-904a-59c4a8cd3816 2024-01-13T09:01:34.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:01:34.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:01:34.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:01:34.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:01:34.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:01:34.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.prepare() 2024-01-13T09:01:34.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:01:34.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:01:34.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:01:34.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:01:34.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:01:34.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:01:34.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:01:34.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:01:34.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:01:34.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T09:01:34.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:01:34.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T09:01:34.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:01:34.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:01:34.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:01:34.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:01:34.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:01:34.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:01:34.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:01:34.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:01:34.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:01:34.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:01:34.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T09:01:34.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:01:34.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:01:34.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:01:34.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T09:01:34.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:01:34.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:01:34.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:01:34.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:01:34.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:01:34.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:01:34.636 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:01:34.636 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:01:34.636 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:01:34.636 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:01:34.636 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:01:34.636 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:01:34.636 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:01:34.636 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:01:34.636 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:01:34.636 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:01:34.637 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:01:34.637 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:01:34.637 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:01:34.637 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:01:34.637 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:01:34.637 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:01:34.637 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:01:34.637 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T09:01:34.637 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:01:34.638 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: main() 2024-01-13T09:01:34.638 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:01:34.638 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: r = ctx.func(ctx) 2024-01-13T09:01:34.638 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:01:34.638 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:01:34.638 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:01:34.638 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:01:34.638 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:01:34.638 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:01:34.639 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:01:34.639 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:01:34.639 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:01:34.639 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:01:34.639 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:01:34.639 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:01:34.639 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4xu2_iuk:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpvn4_lqsj:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:01:34.639 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T09:01:32.277470+0000 mgr.smithi077.amvfyi (mgr.14180) 155 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T09:01:34.639 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: service_id: all-available-devices 2024-01-13T09:01:34.640 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: service_name: osd.all-available-devices 2024-01-13T09:01:34.640 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: placement: 2024-01-13T09:01:34.640 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: host_pattern: '*' 2024-01-13T09:01:34.640 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: spec: 2024-01-13T09:01:34.640 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: data_devices: 2024-01-13T09:01:34.640 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: all: true 2024-01-13T09:01:34.640 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: filter_logic: AND 2024-01-13T09:01:34.640 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: objectstore: bluestore 2024-01-13T09:01:34.641 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T09:01:34.641 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvldeixcm:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpkhpl3aw6:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:01:34.641 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:01:34.641 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:01:34.641 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:01:34.641 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 91ed9908-ebe2-414d-9bf2-3b644218d7c2 2024-01-13T09:01:34.641 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:01:34.641 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:01:34.641 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:01:34.641 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:01:34.642 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:01:34.642 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.prepare() 2024-01-13T09:01:34.642 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:01:34.642 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:01:34.642 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:01:34.642 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:01:34.642 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:01:34.642 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:01:34.642 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:01:34.642 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:01:34.642 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:01:34.643 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T09:01:34.643 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:01:34.644 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T09:01:34.645 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:01:34.645 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:01:34.645 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:01:34.645 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:01:34.645 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:01:34.645 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:01:34.645 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:01:34.645 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:01:34.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:01:34.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:01:34.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T09:01:34.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:01:34.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:01:34.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:01:34.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T09:01:34.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:01:34.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:01:34.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:01:34.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:01:34.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:01:34.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:01:34.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:01:34.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:01:34.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:01:34.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:01:34.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:01:34.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:01:34.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:01:34.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:01:34.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:01:34.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:01:34.648 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:01:34.648 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:01:34.648 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:01:34.648 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:01:34.648 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:01:34.648 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:01:34.648 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:01:34.648 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T09:01:34.648 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:01:34.648 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: main() 2024-01-13T09:01:34.648 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:01:34.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: r = ctx.func(ctx) 2024-01-13T09:01:34.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:01:34.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:01:34.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:01:34.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:01:34.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:01:34.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:01:34.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:01:34.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:01:34.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:01:34.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:01:34.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:01:34.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:01:34.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvldeixcm:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpkhpl3aw6:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:01:34.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T09:01:34.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T09:01:34.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: if self._apply_service(spec): 2024-01-13T09:01:34.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T09:01:34.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T09:01:34.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T09:01:34.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T09:01:34.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T09:01:34.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T09:01:34.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T09:01:34.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T09:01:34.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T09:01:34.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: raise self._value 2024-01-13T09:01:34.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T09:01:34.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: result = (True, func(*args, **kwds)) 2024-01-13T09:01:34.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T09:01:34.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return list(map(*args)) 2024-01-13T09:01:34.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:01:34.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return f(*arg) 2024-01-13T09:01:34.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:01:34.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:01:34.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:01:34.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: code, '\n'.join(err))) 2024-01-13T09:01:34.652 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T09:01:34.652 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvldeixcm:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpkhpl3aw6:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:01:34.652 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:01:34.652 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:01:34.652 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:01:34.652 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 91ed9908-ebe2-414d-9bf2-3b644218d7c2 2024-01-13T09:01:34.652 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:01:34.652 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:01:34.652 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:01:34.652 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:01:34.652 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:01:34.652 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.prepare() 2024-01-13T09:01:34.652 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:01:34.653 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:01:34.653 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:01:34.653 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:01:34.653 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:01:34.653 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:01:34.653 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:01:34.653 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:01:34.653 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:01:34.653 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T09:01:34.653 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:01:34.653 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T09:01:34.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:01:34.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:01:34.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:01:34.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:01:34.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:01:34.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:01:34.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:01:34.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:01:34.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:01:34.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:01:34.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T09:01:34.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:01:34.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:01:34.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:01:34.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T09:01:34.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:01:34.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:01:34.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:01:34.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:01:34.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:01:34.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:01:34.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:01:34.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:01:34.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:01:34.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:01:34.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:01:34.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:01:34.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:01:34.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:01:34.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:01:34.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:01:34.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:01:34.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:01:34.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:01:34.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:01:34.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:01:34.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:01:34.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:01:34.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T09:01:34.657 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:01:34.657 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: main() 2024-01-13T09:01:34.657 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:01:34.657 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: r = ctx.func(ctx) 2024-01-13T09:01:34.657 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:01:34.657 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:01:34.657 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:01:34.657 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:01:34.657 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:01:34.657 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:01:34.657 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:01:34.657 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:01:34.657 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:01:34.657 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:01:34.658 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:01:34.658 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:01:34.658 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpvldeixcm:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpkhpl3aw6:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:01:34.658 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:01:32.278088+0000 mgr.smithi077.amvfyi (mgr.14180) 156 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:34.658 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:01:33.154314+0000 mon.smithi077 (mon.0) 517 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T09:01:35.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:35 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:34.217047+0000 mon.smithi077 (mon.0) 518 : audit [DBG] from='client.? 172.21.15.77:0/2970090742' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:35.594 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:01:35.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:35 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:34.217047+0000 mon.smithi077 (mon.0) 518 : audit [DBG] from='client.? 172.21.15.77:0/2970090742' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:36.459 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:36 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:01:34.278439+0000 mgr.smithi077.amvfyi (mgr.14180) 157 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:36.675 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:36 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:01:34.278439+0000 mgr.smithi077.amvfyi (mgr.14180) 157 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:36.976 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:01:37.343 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:01:38.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:01:36.278987+0000 mgr.smithi077.amvfyi (mgr.14180) 158 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:38.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:36.982453+0000 mon.smithi077 (mon.0) 519 : audit [DBG] from='client.? 172.21.15.77:0/2915490286' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:38.344 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:01:38.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:01:36.278987+0000 mgr.smithi077.amvfyi (mgr.14180) 158 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:38.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:36.982453+0000 mon.smithi077 (mon.0) 519 : audit [DBG] from='client.? 172.21.15.77:0/2915490286' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:39.695 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:01:40.118 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:01:40.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:39 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:01:38.279432+0000 mgr.smithi077.amvfyi (mgr.14180) 159 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:40.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:39 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:39.703335+0000 mon.smithi077 (mon.0) 520 : audit [DBG] from='client.? 172.21.15.77:0/1463156776' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:40.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:39 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:01:38.279432+0000 mgr.smithi077.amvfyi (mgr.14180) 159 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:40.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:39 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:39.703335+0000 mon.smithi077 (mon.0) 520 : audit [DBG] from='client.? 172.21.15.77:0/1463156776' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:41.119 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:01:42.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:41 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:01:40.279768+0000 mgr.smithi077.amvfyi (mgr.14180) 160 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:42.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:41 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:01:40.279768+0000 mgr.smithi077.amvfyi (mgr.14180) 160 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:42.465 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:01:42.858 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:01:43.075 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:42 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:42.472827+0000 mon.smithi077 (mon.0) 521 : audit [DBG] from='client.? 172.21.15.77:0/1765833222' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:43.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:42 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:42.472827+0000 mon.smithi077 (mon.0) 521 : audit [DBG] from='client.? 172.21.15.77:0/1765833222' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:43.859 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:01:44.185 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:43 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:01:42.280173+0000 mgr.smithi077.amvfyi (mgr.14180) 161 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:44.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:44 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:01:42.280173+0000 mgr.smithi077.amvfyi (mgr.14180) 161 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:45.178 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:01:45.558 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:01:46.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:01:44.280483+0000 mgr.smithi077.amvfyi (mgr.14180) 162 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:46.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:45.186158+0000 mon.smithi077 (mon.0) 522 : audit [DBG] from='client.? 172.21.15.77:0/2955163856' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:46.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:46 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:01:44.280483+0000 mgr.smithi077.amvfyi (mgr.14180) 162 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:46.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:46 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:45.186158+0000 mon.smithi077 (mon.0) 522 : audit [DBG] from='client.? 172.21.15.77:0/2955163856' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:46.559 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:01:47.913 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:01:48.248 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:48 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:01:46.281054+0000 mgr.smithi077.amvfyi (mgr.14180) 163 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:48.248 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:48 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:47.920420+0000 mon.smithi077 (mon.0) 523 : audit [DBG] from='client.? 172.21.15.77:0/927758294' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:48.292 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:01:48.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:48 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:01:46.281054+0000 mgr.smithi077.amvfyi (mgr.14180) 163 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:48.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:48 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:47.920420+0000 mon.smithi077 (mon.0) 523 : audit [DBG] from='client.? 172.21.15.77:0/927758294' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:49.293 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:01:50.278 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:50 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:01:48.281635+0000 mgr.smithi077.amvfyi (mgr.14180) 164 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:50.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:50 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:01:48.281635+0000 mgr.smithi077.amvfyi (mgr.14180) 164 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:50.622 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:01:50.999 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:01:51.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:51 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:50.629601+0000 mon.smithi077 (mon.0) 524 : audit [DBG] from='client.? 172.21.15.77:0/1664119016' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:51.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:51 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:50.629601+0000 mon.smithi077 (mon.0) 524 : audit [DBG] from='client.? 172.21.15.77:0/1664119016' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:52.000 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:01:52.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:01:50.282353+0000 mgr.smithi077.amvfyi (mgr.14180) 165 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:52.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:01:50.282353+0000 mgr.smithi077.amvfyi (mgr.14180) 165 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:53.381 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:01:53.791 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:01:54.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:54 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:01:52.282836+0000 mgr.smithi077.amvfyi (mgr.14180) 166 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:54.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:54 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:53.388985+0000 mon.smithi077 (mon.0) 525 : audit [DBG] from='client.? 172.21.15.77:0/435075946' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:54.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:54 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:01:52.282836+0000 mgr.smithi077.amvfyi (mgr.14180) 166 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:54.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:54 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:53.388985+0000 mon.smithi077 (mon.0) 525 : audit [DBG] from='client.? 172.21.15.77:0/435075946' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:54.792 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:01:56.075 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:56 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:01:54.283276+0000 mgr.smithi077.amvfyi (mgr.14180) 167 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:56.159 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:01:56.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:56 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:01:54.283276+0000 mgr.smithi077.amvfyi (mgr.14180) 167 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:56.533 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:01:57.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:57 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:56.166999+0000 mon.smithi077 (mon.0) 526 : audit [DBG] from='client.? 172.21.15.77:0/1095690499' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:57.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:57 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:56.166999+0000 mon.smithi077 (mon.0) 526 : audit [DBG] from='client.? 172.21.15.77:0/1095690499' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:57.534 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:01:58.210 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:58 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:01:56.283689+0000 mgr.smithi077.amvfyi (mgr.14180) 168 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:58.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:58 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:01:56.283689+0000 mgr.smithi077.amvfyi (mgr.14180) 168 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:01:58.913 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:01:59.288 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:01:59 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:01:58.921118+0000 mon.smithi077 (mon.0) 527 : audit [DBG] from='client.? 172.21.15.77:0/456680049' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:01:59.323 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:01:59.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:01:59 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:01:58.921118+0000 mon.smithi077 (mon.0) 527 : audit [DBG] from='client.? 172.21.15.77:0/456680049' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:00.326 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:02:00.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:01:58.284085+0000 mgr.smithi077.amvfyi (mgr.14180) 169 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:00.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:01:58.284085+0000 mgr.smithi077.amvfyi (mgr.14180) 169 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:01.703 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:02:02.040 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:02:02.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:02 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:00.284599+0000 mgr.smithi077.amvfyi (mgr.14180) 170 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:02.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:02 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:01.710046+0000 mon.smithi077 (mon.0) 528 : audit [DBG] from='client.? 172.21.15.77:0/1250186646' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:02.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:02 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:00.284599+0000 mgr.smithi077.amvfyi (mgr.14180) 170 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:02.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:02 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:01.710046+0000 mon.smithi077 (mon.0) 528 : audit [DBG] from='client.? 172.21.15.77:0/1250186646' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:03.042 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:02:04.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:04 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:02.285031+0000 mgr.smithi077.amvfyi (mgr.14180) 171 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:04.351 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:02:04.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:04 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:02.285031+0000 mgr.smithi077.amvfyi (mgr.14180) 171 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:04.747 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:02:05.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:05 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:04.359467+0000 mon.smithi077 (mon.0) 529 : audit [DBG] from='client.? 172.21.15.77:0/2698397082' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:05.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:05 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:04.359467+0000 mon.smithi077 (mon.0) 529 : audit [DBG] from='client.? 172.21.15.77:0/2698397082' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:05.748 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:02:06.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:06 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:04.285433+0000 mgr.smithi077.amvfyi (mgr.14180) 172 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:06.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:06 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:04.285433+0000 mgr.smithi077.amvfyi (mgr.14180) 172 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:07.121 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:02:07.530 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:02:08.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:08 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:06.285819+0000 mgr.smithi077.amvfyi (mgr.14180) 173 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:08.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:08 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:07.129294+0000 mon.smithi077 (mon.0) 530 : audit [DBG] from='client.? 172.21.15.77:0/2778233350' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:08.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:08 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:06.285819+0000 mgr.smithi077.amvfyi (mgr.14180) 173 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:08.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:08 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:07.129294+0000 mon.smithi077 (mon.0) 530 : audit [DBG] from='client.? 172.21.15.77:0/2778233350' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:08.532 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:02:09.910 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:02:10.220 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:10 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:08.286308+0000 mgr.smithi077.amvfyi (mgr.14180) 174 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:10.220 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:10 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:09.917985+0000 mon.smithi077 (mon.0) 531 : audit [DBG] from='client.? 172.21.15.77:0/1501701327' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:10.271 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:02:10.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:10 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:08.286308+0000 mgr.smithi077.amvfyi (mgr.14180) 174 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:10.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:10 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:09.917985+0000 mon.smithi077 (mon.0) 531 : audit [DBG] from='client.? 172.21.15.77:0/1501701327' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:11.272 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:02:12.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:12 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:10.286887+0000 mgr.smithi077.amvfyi (mgr.14180) 175 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:12.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:12 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:10.286887+0000 mgr.smithi077.amvfyi (mgr.14180) 175 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:12.693 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:02:13.055 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:02:13.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:13 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:12.700754+0000 mon.smithi077 (mon.0) 532 : audit [DBG] from='client.? 172.21.15.77:0/2585334857' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:13.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:13 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:12.700754+0000 mon.smithi077 (mon.0) 532 : audit [DBG] from='client.? 172.21.15.77:0/2585334857' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:14.056 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:02:14.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:12.287440+0000 mgr.smithi077.amvfyi (mgr.14180) 176 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:14.405 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:12.287440+0000 mgr.smithi077.amvfyi (mgr.14180) 176 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:15.437 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:02:15.812 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:02:16.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:16 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:14.287827+0000 mgr.smithi077.amvfyi (mgr.14180) 177 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:16.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:16 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:15.444443+0000 mon.smithi077 (mon.0) 533 : audit [DBG] from='client.? 172.21.15.77:0/4077100505' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:16.575 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:16 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:14.287827+0000 mgr.smithi077.amvfyi (mgr.14180) 177 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:16.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:16 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:15.444443+0000 mon.smithi077 (mon.0) 533 : audit [DBG] from='client.? 172.21.15.77:0/4077100505' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:16.813 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:02:18.176 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:02:18.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:18 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:16.288458+0000 mgr.smithi077.amvfyi (mgr.14180) 178 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:18.390 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:18 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:16.288458+0000 mgr.smithi077.amvfyi (mgr.14180) 178 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:18.580 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:02:19.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:19 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:18.184489+0000 mon.smithi077 (mon.0) 534 : audit [DBG] from='client.? 172.21.15.77:0/3373559002' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:19.575 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:19 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:18.184489+0000 mon.smithi077 (mon.0) 534 : audit [DBG] from='client.? 172.21.15.77:0/3373559002' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:19.581 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:02:20.257 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:20 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:18.288912+0000 mgr.smithi077.amvfyi (mgr.14180) 179 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:20.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:20 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:18.288912+0000 mgr.smithi077.amvfyi (mgr.14180) 179 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:20.935 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:02:21.269 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:20.943602+0000 mon.smithi077 (mon.0) 535 : audit [DBG] from='client.? 172.21.15.77:0/3425622346' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:21.295 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:02:21.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:20.943602+0000 mon.smithi077 (mon.0) 535 : audit [DBG] from='client.? 172.21.15.77:0/3425622346' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:22.296 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:02:22.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:20.289577+0000 mgr.smithi077.amvfyi (mgr.14180) 180 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:22.575 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:20.289577+0000 mgr.smithi077.amvfyi (mgr.14180) 180 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:23.588 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:02:23.970 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:02:24.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:24 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:22.290204+0000 mgr.smithi077.amvfyi (mgr.14180) 181 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:24.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:24 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:23.596499+0000 mon.smithi077 (mon.0) 536 : audit [DBG] from='client.? 172.21.15.77:0/1589065865' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:24.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:24 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:22.290204+0000 mgr.smithi077.amvfyi (mgr.14180) 181 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:24.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:24 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:23.596499+0000 mon.smithi077 (mon.0) 536 : audit [DBG] from='client.? 172.21.15.77:0/1589065865' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:24.971 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:02:26.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:26 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:24.290606+0000 mgr.smithi077.amvfyi (mgr.14180) 182 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:26.333 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:02:26.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:26 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:24.290606+0000 mgr.smithi077.amvfyi (mgr.14180) 182 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:26.728 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:02:27.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:27 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:26.341538+0000 mon.smithi077 (mon.0) 537 : audit [DBG] from='client.? 172.21.15.77:0/1548501678' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:27.575 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:27 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:26.341538+0000 mon.smithi077 (mon.0) 537 : audit [DBG] from='client.? 172.21.15.77:0/1548501678' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:27.729 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:02:28.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:28 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:26.290999+0000 mgr.smithi077.amvfyi (mgr.14180) 183 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:28.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:28 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:26.290999+0000 mgr.smithi077.amvfyi (mgr.14180) 183 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:29.041 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:02:29.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:29.049350+0000 mon.smithi077 (mon.0) 538 : audit [DBG] from='client.? 172.21.15.77:0/325469183' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:29.401 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:02:29.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:29.049350+0000 mon.smithi077 (mon.0) 538 : audit [DBG] from='client.? 172.21.15.77:0/325469183' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:30.402 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:02:30.435 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:30 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:28.291388+0000 mgr.smithi077.amvfyi (mgr.14180) 184 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:30.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:30 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:28.291388+0000 mgr.smithi077.amvfyi (mgr.14180) 184 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:31.724 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:02:32.095 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:02:32.575 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:32 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:30.291758+0000 mgr.smithi077.amvfyi (mgr.14180) 185 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:32.575 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:32 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:31.731421+0000 mon.smithi077 (mon.0) 539 : audit [DBG] from='client.? 172.21.15.77:0/1085319923' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:32.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:30.291758+0000 mgr.smithi077.amvfyi (mgr.14180) 185 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:32.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:31.731421+0000 mon.smithi077 (mon.0) 539 : audit [DBG] from='client.? 172.21.15.77:0/1085319923' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:33.096 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:02:33.234 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:32.279632+0000 mon.smithi077 (mon.0) 540 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T09:02:33.234 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:32.575321+0000 mon.smithi077 (mon.0) 541 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:02:33.235 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:32.575897+0000 mon.smithi077 (mon.0) 542 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T09:02:33.235 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:32.580115+0000 mon.smithi077 (mon.0) 543 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T09:02:33.235 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:32.583440+0000 mon.smithi077 (mon.0) 544 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T09:02:33.235 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:32.586067+0000 mon.smithi077 (mon.0) 545 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:02:33.235 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:32.587060+0000 mon.smithi077 (mon.0) 546 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:02:33.235 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:32.587253+0000 mon.smithi077 (mon.0) 547 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:02:33.235 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:32.587647+0000 mon.smithi077 (mon.0) 548 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:02:33.248 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:32.279632+0000 mon.smithi077 (mon.0) 540 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T09:02:33.248 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:32.575321+0000 mon.smithi077 (mon.0) 541 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:02:33.248 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:32.575897+0000 mon.smithi077 (mon.0) 542 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T09:02:33.248 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:32.580115+0000 mon.smithi077 (mon.0) 543 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T09:02:33.248 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:32.583440+0000 mon.smithi077 (mon.0) 544 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T09:02:33.248 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:32.586067+0000 mon.smithi077 (mon.0) 545 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:02:33.248 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:32.587060+0000 mon.smithi077 (mon.0) 546 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:02:33.248 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:32.587253+0000 mon.smithi077 (mon.0) 547 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:02:33.248 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:32.587647+0000 mon.smithi077 (mon.0) 548 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:02:34.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:32.292154+0000 mgr.smithi077.amvfyi (mgr.14180) 186 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:34.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:32.580436+0000 mgr.smithi077.amvfyi (mgr.14180) 187 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:34.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:33.149796+0000 mon.smithi077 (mon.0) 549 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T09:02:34.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:34 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:33.149823+0000 mon.smithi077 (mon.0) 550 : cluster [INF] Cluster is now healthy 2024-01-13T09:02:34.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:32.292154+0000 mgr.smithi077.amvfyi (mgr.14180) 186 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:34.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:32.580436+0000 mgr.smithi077.amvfyi (mgr.14180) 187 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:34.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:33.149796+0000 mon.smithi077 (mon.0) 549 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T09:02:34.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:34 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:33.149823+0000 mon.smithi077 (mon.0) 550 : cluster [INF] Cluster is now healthy 2024-01-13T09:02:34.711 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:02:35.087 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":26,"num_osds":1,"num_up_osds":0,"osd_up_since":0,"num_in_osds":1,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:02:35.306 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:35 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:34.657515+0000 mon.smithi141 (mon.1) 13 : audit [INF] from='client.? 172.21.15.141:0/1612758994' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "742cf295-4873-4101-a894-53b1d93e4afa"}]: dispatch 2024-01-13T09:02:35.306 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:35 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:34.657939+0000 mon.smithi077 (mon.0) 551 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "742cf295-4873-4101-a894-53b1d93e4afa"}]: dispatch 2024-01-13T09:02:35.306 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:35 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:34.662524+0000 mon.smithi077 (mon.0) 552 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "742cf295-4873-4101-a894-53b1d93e4afa"}]': finished 2024-01-13T09:02:35.306 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:35 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:34.662594+0000 mon.smithi077 (mon.0) 553 : cluster [DBG] osdmap e26: 1 total, 0 up, 1 in 2024-01-13T09:02:35.306 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:35 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:34.662697+0000 mon.smithi077 (mon.0) 554 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:02:35.306 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:35 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:34.719307+0000 mon.smithi077 (mon.0) 555 : audit [DBG] from='client.? 172.21.15.77:0/1437791789' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:35.306 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:35 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:34.725375+0000 mon.smithi077 (mon.0) 556 : audit [INF] from='client.? 172.21.15.77:0/1681182300' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "380ef499-bd42-42ae-bc9d-5cebe43a9089"}]: dispatch 2024-01-13T09:02:35.306 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:35 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:34.729898+0000 mon.smithi077 (mon.0) 557 : audit [INF] from='client.? 172.21.15.77:0/1681182300' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "380ef499-bd42-42ae-bc9d-5cebe43a9089"}]': finished 2024-01-13T09:02:35.306 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:35 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:34.729959+0000 mon.smithi077 (mon.0) 558 : cluster [DBG] osdmap e27: 2 total, 0 up, 2 in 2024-01-13T09:02:35.306 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:35 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:34.730088+0000 mon.smithi077 (mon.0) 559 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:02:35.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:35 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:34.730429+0000 mon.smithi077 (mon.0) 560 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-13T09:02:35.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:35 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:34.955378+0000 mon.smithi141 (mon.1) 14 : audit [INF] from='client.? 172.21.15.141:0/1507066146' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:02:35.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:35 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:34.955723+0000 mon.smithi077 (mon.0) 561 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:02:35.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:35 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:34.959066+0000 mon.smithi077 (mon.0) 562 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:02:35.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:35 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:34.959113+0000 mon.smithi077 (mon.0) 563 : cluster [DBG] osdmap e28: 1 total, 0 up, 1 in 2024-01-13T09:02:35.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:35 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:34.959211+0000 mon.smithi077 (mon.0) 564 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-13T09:02:35.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:35 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:35.030365+0000 mon.smithi077 (mon.0) 565 : audit [INF] from='client.? 172.21.15.77:0/2039547198' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:02:35.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:35 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:35.033893+0000 mon.smithi077 (mon.0) 566 : audit [INF] from='client.? 172.21.15.77:0/2039547198' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:02:35.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:35 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:35.033934+0000 mon.smithi077 (mon.0) 567 : cluster [DBG] osdmap e29: 0 total, 0 up, 0 in 2024-01-13T09:02:35.312 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:35 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:34.657515+0000 mon.smithi141 (mon.1) 13 : audit [INF] from='client.? 172.21.15.141:0/1612758994' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "742cf295-4873-4101-a894-53b1d93e4afa"}]: dispatch 2024-01-13T09:02:35.312 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:35 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:34.657939+0000 mon.smithi077 (mon.0) 551 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "742cf295-4873-4101-a894-53b1d93e4afa"}]: dispatch 2024-01-13T09:02:35.312 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:35 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:34.662524+0000 mon.smithi077 (mon.0) 552 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "742cf295-4873-4101-a894-53b1d93e4afa"}]': finished 2024-01-13T09:02:35.312 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:35 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:34.662594+0000 mon.smithi077 (mon.0) 553 : cluster [DBG] osdmap e26: 1 total, 0 up, 1 in 2024-01-13T09:02:35.312 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:35 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:34.662697+0000 mon.smithi077 (mon.0) 554 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:02:35.312 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:35 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:34.719307+0000 mon.smithi077 (mon.0) 555 : audit [DBG] from='client.? 172.21.15.77:0/1437791789' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:35.313 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:35 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:34.725375+0000 mon.smithi077 (mon.0) 556 : audit [INF] from='client.? 172.21.15.77:0/1681182300' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "380ef499-bd42-42ae-bc9d-5cebe43a9089"}]: dispatch 2024-01-13T09:02:35.313 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:35 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:34.729898+0000 mon.smithi077 (mon.0) 557 : audit [INF] from='client.? 172.21.15.77:0/1681182300' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "380ef499-bd42-42ae-bc9d-5cebe43a9089"}]': finished 2024-01-13T09:02:35.313 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:35 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:34.729959+0000 mon.smithi077 (mon.0) 558 : cluster [DBG] osdmap e27: 2 total, 0 up, 2 in 2024-01-13T09:02:35.313 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:35 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:34.730088+0000 mon.smithi077 (mon.0) 559 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:02:35.313 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:35 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:34.730429+0000 mon.smithi077 (mon.0) 560 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-13T09:02:35.313 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:35 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:34.955378+0000 mon.smithi141 (mon.1) 14 : audit [INF] from='client.? 172.21.15.141:0/1507066146' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:02:35.313 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:35 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:34.955723+0000 mon.smithi077 (mon.0) 561 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:02:35.313 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:35 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:34.959066+0000 mon.smithi077 (mon.0) 562 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:02:35.313 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:35 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:34.959113+0000 mon.smithi077 (mon.0) 563 : cluster [DBG] osdmap e28: 1 total, 0 up, 1 in 2024-01-13T09:02:35.313 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:35 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:34.959211+0000 mon.smithi077 (mon.0) 564 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-13T09:02:35.313 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:35 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:35.030365+0000 mon.smithi077 (mon.0) 565 : audit [INF] from='client.? 172.21.15.77:0/2039547198' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:02:35.313 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:35 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:35.033893+0000 mon.smithi077 (mon.0) 566 : audit [INF] from='client.? 172.21.15.77:0/2039547198' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:02:35.313 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:35 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:35.033934+0000 mon.smithi077 (mon.0) 567 : cluster [DBG] osdmap e29: 0 total, 0 up, 0 in 2024-01-13T09:02:36.088 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:02:36.471 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:36 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:34.580637+0000 mgr.smithi077.amvfyi (mgr.14180) 188 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:36.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:36 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:34.580637+0000 mgr.smithi077.amvfyi (mgr.14180) 188 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:37.463 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:02:37.491 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T09:02:35.430052+0000 mgr.smithi077.amvfyi (mgr.14180) 189 : cephadm [ERR] executing create_from_spec_one(([('smithi077', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi141', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:02:37.492 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T09:02:37.492 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:02:37.492 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return f(*arg) 2024-01-13T09:02:37.492 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:02:37.492 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:02:37.492 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:02:37.492 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: code, '\n'.join(err))) 2024-01-13T09:02:37.492 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T09:02:37.492 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp7i86qhz1:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp450yqn5o:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:02:37.492 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:02:37.492 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:02:37.492 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:02:37.492 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 742cf295-4873-4101-a894-53b1d93e4afa 2024-01-13T09:02:37.493 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:02:37.493 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:02:37.493 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:02:37.493 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:02:37.493 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:02:37.493 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.prepare() 2024-01-13T09:02:37.493 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:02:37.493 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:02:37.493 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:02:37.493 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:02:37.493 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:02:37.493 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:02:37.493 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:02:37.493 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:02:37.493 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:02:37.494 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T09:02:37.494 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:02:37.494 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T09:02:37.494 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:02:37.494 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:02:37.494 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:02:37.494 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:02:37.494 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:02:37.494 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:02:37.494 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:02:37.494 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:02:37.494 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:02:37.494 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:02:37.494 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T09:02:37.494 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:02:37.495 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:02:37.495 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:02:37.495 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T09:02:37.495 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:02:37.495 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:02:37.495 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:02:37.495 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:02:37.495 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:02:37.495 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:02:37.495 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:02:37.496 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:02:37.496 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:02:37.496 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:02:37.496 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:02:37.496 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:02:37.496 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:02:37.496 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:02:37.496 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:02:37.496 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:02:37.496 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:02:37.496 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:02:37.496 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:02:37.496 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:02:37.496 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:02:37.497 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:02:37.497 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:02:37.497 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T09:02:37.497 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:02:37.497 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: main() 2024-01-13T09:02:37.497 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:02:37.497 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: r = ctx.func(ctx) 2024-01-13T09:02:37.497 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:02:37.497 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:02:37.497 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:02:37.497 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:02:37.497 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:02:37.497 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:02:37.497 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:02:37.497 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:02:37.498 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:02:37.498 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:02:37.498 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:02:37.498 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:02:37.498 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp7i86qhz1:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp450yqn5o:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:02:37.498 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T09:02:35.493289+0000 mgr.smithi077.amvfyi (mgr.14180) 190 : cephadm [ERR] executing create_from_spec_one(([('smithi077', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi141', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:02:37.498 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T09:02:37.498 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:02:37.498 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return f(*arg) 2024-01-13T09:02:37.498 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:02:37.498 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:02:37.498 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:02:37.498 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: code, '\n'.join(err))) 2024-01-13T09:02:37.498 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi077/config 2024-01-13T09:02:37.498 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmproq_pzmc:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpqr6y3enp:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:02:37.499 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:02:37.499 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:02:37.499 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:02:37.499 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 380ef499-bd42-42ae-bc9d-5cebe43a9089 2024-01-13T09:02:37.499 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:02:37.499 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 --yes-i-really-mean-it 2024-01-13T09:02:37.499 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr stderr: purged osd.1 2024-01-13T09:02:37.499 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:02:37.499 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:02:37.499 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.prepare() 2024-01-13T09:02:37.499 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:02:37.499 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:02:37.499 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:02:37.499 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:02:37.500 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:02:37.500 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:02:37.500 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:02:37.500 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:02:37.500 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:02:37.500 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T09:02:37.500 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:02:37.500 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T09:02:37.500 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:02:37.500 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:02:37.500 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:02:37.500 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:02:37.500 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:02:37.500 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:02:37.501 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:02:37.501 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:02:37.501 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:02:37.501 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:02:37.501 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T09:02:37.501 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:02:37.501 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:02:37.501 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:02:37.501 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T09:02:37.501 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:02:37.501 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:02:37.501 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:02:37.501 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:02:37.501 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:02:37.501 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:02:37.502 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:02:37.502 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:02:37.502 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:02:37.502 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:02:37.502 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:02:37.502 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:02:37.502 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:02:37.502 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:02:37.502 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:02:37.502 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:02:37.502 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:02:37.502 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:02:37.502 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:02:37.502 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:02:37.502 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:02:37.502 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:02:37.503 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 1 2024-01-13T09:02:37.503 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T09:02:37.503 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:02:37.503 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: main() 2024-01-13T09:02:37.503 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:02:37.503 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: r = ctx.func(ctx) 2024-01-13T09:02:37.503 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:02:37.503 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:02:37.503 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:02:37.503 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:02:37.503 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:02:37.503 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:02:37.503 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:02:37.503 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:02:37.504 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:02:37.504 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:02:37.504 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:02:37.504 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:02:37.504 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmproq_pzmc:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpqr6y3enp:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:02:37.504 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T09:02:35.494648+0000 mgr.smithi077.amvfyi (mgr.14180) 191 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T09:02:37.504 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: service_id: all-available-devices 2024-01-13T09:02:37.504 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: service_name: osd.all-available-devices 2024-01-13T09:02:37.504 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: placement: 2024-01-13T09:02:37.504 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: host_pattern: '*' 2024-01-13T09:02:37.504 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: spec: 2024-01-13T09:02:37.504 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: data_devices: 2024-01-13T09:02:37.506 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: all: true 2024-01-13T09:02:37.506 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: filter_logic: AND 2024-01-13T09:02:37.506 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: objectstore: bluestore 2024-01-13T09:02:37.506 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T09:02:37.506 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp7i86qhz1:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp450yqn5o:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:02:37.506 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:02:37.506 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:02:37.506 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:02:37.506 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 742cf295-4873-4101-a894-53b1d93e4afa 2024-01-13T09:02:37.506 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:02:37.506 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:02:37.507 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:02:37.507 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:02:37.507 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:02:37.507 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.prepare() 2024-01-13T09:02:37.507 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:02:37.507 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:02:37.507 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:02:37.507 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:02:37.507 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:02:37.507 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:02:37.507 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:02:37.507 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:02:37.507 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:02:37.507 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T09:02:37.507 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:02:37.508 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T09:02:37.508 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:02:37.508 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:02:37.508 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:02:37.508 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:02:37.508 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:02:37.508 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:02:37.508 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:02:37.508 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:02:37.508 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:02:37.508 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:02:37.508 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T09:02:37.508 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:02:37.508 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:02:37.508 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:02:37.509 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T09:02:37.509 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:02:37.509 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:02:37.509 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:02:37.509 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:02:37.509 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:02:37.509 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:02:37.509 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:02:37.509 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:02:37.509 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:02:37.509 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:02:37.509 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:02:37.509 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:02:37.509 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:02:37.510 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:02:37.510 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:02:37.510 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:02:37.510 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:02:37.510 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:02:37.510 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:02:37.510 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:02:37.510 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:02:37.510 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:02:37.510 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:02:37.510 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T09:02:37.510 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:02:37.510 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: main() 2024-01-13T09:02:37.510 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:02:37.510 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: r = ctx.func(ctx) 2024-01-13T09:02:37.511 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:02:37.511 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:02:37.511 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:02:37.511 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:02:37.511 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:02:37.511 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:02:37.511 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:02:37.511 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:02:37.511 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:02:37.511 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:02:37.511 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:02:37.511 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:02:37.511 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp7i86qhz1:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp450yqn5o:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:02:37.511 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T09:02:37.511 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T09:02:37.512 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: if self._apply_service(spec): 2024-01-13T09:02:37.512 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T09:02:37.512 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T09:02:37.512 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T09:02:37.512 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T09:02:37.512 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T09:02:37.512 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T09:02:37.512 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T09:02:37.512 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T09:02:37.512 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T09:02:37.512 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: raise self._value 2024-01-13T09:02:37.512 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T09:02:37.512 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: result = (True, func(*args, **kwds)) 2024-01-13T09:02:37.513 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T09:02:37.513 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return list(map(*args)) 2024-01-13T09:02:37.513 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:02:37.513 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return f(*arg) 2024-01-13T09:02:37.513 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:02:37.513 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:02:37.513 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:02:37.513 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: code, '\n'.join(err))) 2024-01-13T09:02:37.513 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T09:02:37.513 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp7i86qhz1:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp450yqn5o:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:02:37.513 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:02:37.513 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:02:37.513 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:02:37.513 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 742cf295-4873-4101-a894-53b1d93e4afa 2024-01-13T09:02:37.513 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:02:37.514 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:02:37.514 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:02:37.514 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:02:37.514 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:02:37.514 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.prepare() 2024-01-13T09:02:37.514 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:02:37.514 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:02:37.514 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:02:37.514 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:02:37.514 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:02:37.514 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:02:37.514 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:02:37.514 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:02:37.514 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:02:37.515 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T09:02:37.515 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:02:37.515 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T09:02:37.515 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:02:37.515 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:02:37.515 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:02:37.515 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:02:37.515 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:02:37.515 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:02:37.515 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:02:37.515 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:02:37.515 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:02:37.515 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:02:37.515 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T09:02:37.515 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:02:37.516 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:02:37.516 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:02:37.516 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T09:02:37.516 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:02:37.516 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:02:37.516 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:02:37.516 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:02:37.516 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:02:37.516 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:02:37.516 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:02:37.516 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:02:37.516 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:02:37.516 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:02:37.516 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:02:37.516 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:02:37.517 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:02:37.517 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:02:37.517 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:02:37.517 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:02:37.517 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:02:37.517 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:02:37.517 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:02:37.517 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:02:37.517 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:02:37.517 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:02:37.517 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:02:37.517 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T09:02:37.517 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:02:37.517 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: main() 2024-01-13T09:02:37.518 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:02:37.518 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: r = ctx.func(ctx) 2024-01-13T09:02:37.518 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:02:37.518 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:02:37.518 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:02:37.518 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:02:37.518 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:02:37.518 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:02:37.518 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:02:37.518 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:02:37.518 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:02:37.518 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:02:37.518 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:02:37.518 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:02:37.518 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp7i86qhz1:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp450yqn5o:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:02:37.519 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:35.495294+0000 mgr.smithi077.amvfyi (mgr.14180) 192 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:37.519 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:36.167068+0000 mon.smithi077 (mon.0) 568 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T09:02:37.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T09:02:35.430052+0000 mgr.smithi077.amvfyi (mgr.14180) 189 : cephadm [ERR] executing create_from_spec_one(([('smithi077', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi141', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:02:37.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T09:02:37.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:02:37.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return f(*arg) 2024-01-13T09:02:37.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:02:37.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:02:37.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:02:37.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: code, '\n'.join(err))) 2024-01-13T09:02:37.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T09:02:37.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp7i86qhz1:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp450yqn5o:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:02:37.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:02:37.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:02:37.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:02:37.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 742cf295-4873-4101-a894-53b1d93e4afa 2024-01-13T09:02:37.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:02:37.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:02:37.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:02:37.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:02:37.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:02:37.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.prepare() 2024-01-13T09:02:37.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:02:37.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:02:37.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:02:37.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:02:37.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:02:37.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:02:37.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:02:37.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:02:37.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:02:37.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T09:02:37.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:02:37.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T09:02:37.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:02:37.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:02:37.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:02:37.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:02:37.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:02:37.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:02:37.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:02:37.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:02:37.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:02:37.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:02:37.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T09:02:37.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:02:37.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:02:37.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:02:37.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T09:02:37.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:02:37.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:02:37.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:02:37.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:02:37.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:02:37.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:02:37.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:02:37.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:02:37.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:02:37.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:02:37.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:02:37.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:02:37.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:02:37.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:02:37.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:02:37.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:02:37.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:02:37.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:02:37.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:02:37.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:02:37.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:02:37.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:02:37.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:02:37.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T09:02:37.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:02:37.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: main() 2024-01-13T09:02:37.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:02:37.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: r = ctx.func(ctx) 2024-01-13T09:02:37.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:02:37.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:02:37.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:02:37.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:02:37.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:02:37.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:02:37.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:02:37.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:02:37.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:02:37.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:02:37.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:02:37.636 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:02:37.636 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp7i86qhz1:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp450yqn5o:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:02:37.636 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T09:02:35.493289+0000 mgr.smithi077.amvfyi (mgr.14180) 190 : cephadm [ERR] executing create_from_spec_one(([('smithi077', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi141', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:02:37.636 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T09:02:37.636 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:02:37.636 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return f(*arg) 2024-01-13T09:02:37.636 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:02:37.636 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:02:37.636 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:02:37.636 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: code, '\n'.join(err))) 2024-01-13T09:02:37.636 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi077/config 2024-01-13T09:02:37.636 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmproq_pzmc:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpqr6y3enp:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:02:37.636 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:02:37.637 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:02:37.637 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:02:37.637 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 380ef499-bd42-42ae-bc9d-5cebe43a9089 2024-01-13T09:02:37.637 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:02:37.637 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 --yes-i-really-mean-it 2024-01-13T09:02:37.637 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr stderr: purged osd.1 2024-01-13T09:02:37.637 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:02:37.637 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:02:37.637 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.prepare() 2024-01-13T09:02:37.637 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:02:37.637 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:02:37.637 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:02:37.637 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:02:37.637 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:02:37.637 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:02:37.638 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:02:37.638 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:02:37.638 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:02:37.638 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T09:02:37.638 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:02:37.638 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T09:02:37.638 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:02:37.638 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:02:37.638 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:02:37.638 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:02:37.638 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:02:37.638 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:02:37.638 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:02:37.638 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:02:37.638 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:02:37.639 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:02:37.639 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T09:02:37.639 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:02:37.639 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:02:37.639 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:02:37.639 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T09:02:37.639 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:02:37.639 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:02:37.639 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:02:37.639 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:02:37.639 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:02:37.639 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:02:37.639 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:02:37.639 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:02:37.639 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:02:37.640 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:02:37.640 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:02:37.640 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:02:37.640 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:02:37.640 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:02:37.640 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:02:37.640 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:02:37.640 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:02:37.640 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:02:37.640 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:02:37.640 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:02:37.640 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:02:37.640 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:02:37.640 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 1 2024-01-13T09:02:37.640 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T09:02:37.641 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:02:37.641 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: main() 2024-01-13T09:02:37.641 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:02:37.641 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: r = ctx.func(ctx) 2024-01-13T09:02:37.641 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:02:37.641 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:02:37.642 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:02:37.642 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:02:37.643 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:02:37.643 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:02:37.643 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:02:37.643 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:02:37.643 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:02:37.643 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:02:37.643 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:02:37.643 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:02:37.643 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmproq_pzmc:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpqr6y3enp:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:02:37.643 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T09:02:35.494648+0000 mgr.smithi077.amvfyi (mgr.14180) 191 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T09:02:37.643 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: service_id: all-available-devices 2024-01-13T09:02:37.643 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: service_name: osd.all-available-devices 2024-01-13T09:02:37.643 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: placement: 2024-01-13T09:02:37.643 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: host_pattern: '*' 2024-01-13T09:02:37.644 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: spec: 2024-01-13T09:02:37.644 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: data_devices: 2024-01-13T09:02:37.644 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: all: true 2024-01-13T09:02:37.644 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: filter_logic: AND 2024-01-13T09:02:37.644 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: objectstore: bluestore 2024-01-13T09:02:37.644 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T09:02:37.644 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp7i86qhz1:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp450yqn5o:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:02:37.644 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:02:37.644 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:02:37.644 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:02:37.644 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 742cf295-4873-4101-a894-53b1d93e4afa 2024-01-13T09:02:37.644 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:02:37.644 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:02:37.644 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:02:37.645 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:02:37.645 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:02:37.645 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.prepare() 2024-01-13T09:02:37.645 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:02:37.645 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:02:37.645 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:02:37.645 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:02:37.645 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:02:37.645 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:02:37.645 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:02:37.645 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:02:37.645 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:02:37.645 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T09:02:37.645 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:02:37.645 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T09:02:37.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:02:37.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:02:37.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:02:37.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:02:37.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:02:37.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:02:37.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:02:37.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:02:37.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:02:37.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:02:37.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T09:02:37.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:02:37.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:02:37.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:02:37.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T09:02:37.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:02:37.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:02:37.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:02:37.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:02:37.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:02:37.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:02:37.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:02:37.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:02:37.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:02:37.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:02:37.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:02:37.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:02:37.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:02:37.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:02:37.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:02:37.648 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:02:37.648 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:02:37.648 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:02:37.648 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:02:37.648 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:02:37.648 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:02:37.648 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:02:37.648 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:02:37.648 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T09:02:37.648 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:02:37.648 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: main() 2024-01-13T09:02:37.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:02:37.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: r = ctx.func(ctx) 2024-01-13T09:02:37.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:02:37.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:02:37.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:02:37.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:02:37.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:02:37.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:02:37.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:02:37.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:02:37.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:02:37.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:02:37.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:02:37.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:02:37.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp7i86qhz1:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp450yqn5o:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:02:37.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T09:02:37.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T09:02:37.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: if self._apply_service(spec): 2024-01-13T09:02:37.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T09:02:37.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T09:02:37.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T09:02:37.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T09:02:37.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T09:02:37.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T09:02:37.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T09:02:37.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T09:02:37.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T09:02:37.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: raise self._value 2024-01-13T09:02:37.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T09:02:37.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: result = (True, func(*args, **kwds)) 2024-01-13T09:02:37.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T09:02:37.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return list(map(*args)) 2024-01-13T09:02:37.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:02:37.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return f(*arg) 2024-01-13T09:02:37.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:02:37.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:02:37.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:02:37.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: code, '\n'.join(err))) 2024-01-13T09:02:37.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T09:02:37.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp7i86qhz1:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp450yqn5o:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:02:37.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:02:37.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:02:37.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:02:37.652 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 742cf295-4873-4101-a894-53b1d93e4afa 2024-01-13T09:02:37.652 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:02:37.652 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:02:37.652 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:02:37.652 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:02:37.652 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:02:37.652 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.prepare() 2024-01-13T09:02:37.652 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:02:37.652 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:02:37.652 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:02:37.652 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:02:37.652 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:02:37.652 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:02:37.652 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:02:37.653 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:02:37.653 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:02:37.653 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T09:02:37.653 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:02:37.653 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T09:02:37.653 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:02:37.653 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:02:37.653 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:02:37.653 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:02:37.653 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:02:37.653 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:02:37.653 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:02:37.653 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:02:37.653 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:02:37.653 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:02:37.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T09:02:37.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:02:37.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:02:37.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:02:37.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T09:02:37.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:02:37.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:02:37.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:02:37.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:02:37.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:02:37.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:02:37.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:02:37.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:02:37.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:02:37.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:02:37.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:02:37.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:02:37.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:02:37.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:02:37.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:02:37.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:02:37.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:02:37.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:02:37.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:02:37.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:02:37.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:02:37.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:02:37.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:02:37.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T09:02:37.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:02:37.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: main() 2024-01-13T09:02:37.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:02:37.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: r = ctx.func(ctx) 2024-01-13T09:02:37.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:02:37.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:02:37.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:02:37.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:02:37.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:02:37.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:02:37.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:02:37.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:02:37.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:02:37.657 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:02:37.657 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:02:37.657 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:02:37.657 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp7i86qhz1:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp450yqn5o:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:02:37.657 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:35.495294+0000 mgr.smithi077.amvfyi (mgr.14180) 192 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:37.657 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:36.167068+0000 mon.smithi077 (mon.0) 568 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T09:02:37.906 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:02:38.575 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:38 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:37.470535+0000 mon.smithi077 (mon.0) 569 : audit [DBG] from='client.? 172.21.15.77:0/2557024936' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:38.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:38 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:37.470535+0000 mon.smithi077 (mon.0) 569 : audit [DBG] from='client.? 172.21.15.77:0/2557024936' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:38.908 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:02:39.237 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:39 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:37.495835+0000 mgr.smithi077.amvfyi (mgr.14180) 193 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:39.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:39 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:37.495835+0000 mgr.smithi077.amvfyi (mgr.14180) 193 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:40.225 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:02:40.602 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:02:41.575 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:41 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:39.496202+0000 mgr.smithi077.amvfyi (mgr.14180) 194 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:41.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:41 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:40.233179+0000 mon.smithi077 (mon.0) 570 : audit [DBG] from='client.? 172.21.15.77:0/108710284' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:41.603 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:02:41.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:41 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:39.496202+0000 mgr.smithi077.amvfyi (mgr.14180) 194 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:41.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:41 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:40.233179+0000 mon.smithi077 (mon.0) 570 : audit [DBG] from='client.? 172.21.15.77:0/108710284' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:42.907 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:02:43.284 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:02:43.575 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:43 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:41.496755+0000 mgr.smithi077.amvfyi (mgr.14180) 195 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:43.575 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:43 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:42.915138+0000 mon.smithi077 (mon.0) 571 : audit [DBG] from='client.? 172.21.15.77:0/3578972404' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:43.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:43 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:41.496755+0000 mgr.smithi077.amvfyi (mgr.14180) 195 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:43.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:43 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:42.915138+0000 mon.smithi077 (mon.0) 571 : audit [DBG] from='client.? 172.21.15.77:0/3578972404' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:44.285 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:02:45.272 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:45 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:43.497183+0000 mgr.smithi077.amvfyi (mgr.14180) 196 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:45.628 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:45 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:43.497183+0000 mgr.smithi077.amvfyi (mgr.14180) 196 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:45.642 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:02:46.075 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:02:46.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:45.497700+0000 mgr.smithi077.amvfyi (mgr.14180) 197 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:46.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:45.651008+0000 mon.smithi077 (mon.0) 572 : audit [DBG] from='client.? 172.21.15.77:0/2793480403' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:46.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:46 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:45.497700+0000 mgr.smithi077.amvfyi (mgr.14180) 197 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:46.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:46 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:45.651008+0000 mon.smithi077 (mon.0) 572 : audit [DBG] from='client.? 172.21.15.77:0/2793480403' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:47.076 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:02:48.405 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:02:48.768 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:02:49.075 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:49 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:47.498487+0000 mgr.smithi077.amvfyi (mgr.14180) 198 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:49.076 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:49 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:48.413182+0000 mon.smithi077 (mon.0) 573 : audit [DBG] from='client.? 172.21.15.77:0/3239503678' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:49.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:49 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:47.498487+0000 mgr.smithi077.amvfyi (mgr.14180) 198 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:49.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:49 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:48.413182+0000 mon.smithi077 (mon.0) 573 : audit [DBG] from='client.? 172.21.15.77:0/3239503678' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:49.769 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:02:51.075 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:51 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:49.499033+0000 mgr.smithi077.amvfyi (mgr.14180) 199 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:51.098 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:02:51.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:51 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:49.499033+0000 mgr.smithi077.amvfyi (mgr.14180) 199 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:51.475 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:02:52.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:51.107039+0000 mon.smithi077 (mon.0) 574 : audit [DBG] from='client.? 172.21.15.77:0/3379815495' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:52.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:51.107039+0000 mon.smithi077 (mon.0) 574 : audit [DBG] from='client.? 172.21.15.77:0/3379815495' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:52.476 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:02:53.184 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:51.499444+0000 mgr.smithi077.amvfyi (mgr.14180) 200 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:53.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:51.499444+0000 mgr.smithi077.amvfyi (mgr.14180) 200 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:53.840 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:02:54.198 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:54 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:53.848592+0000 mon.smithi077 (mon.0) 575 : audit [DBG] from='client.? 172.21.15.77:0/2907704168' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:54.258 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:02:54.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:54 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:53.848592+0000 mon.smithi077 (mon.0) 575 : audit [DBG] from='client.? 172.21.15.77:0/2907704168' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:55.259 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:02:55.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:55 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:53.499796+0000 mgr.smithi077.amvfyi (mgr.14180) 201 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:55.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:55 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:53.499796+0000 mgr.smithi077.amvfyi (mgr.14180) 201 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:56.564 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:02:56.932 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:02:57.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:57 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:55.500353+0000 mgr.smithi077.amvfyi (mgr.14180) 202 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:57.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:57 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:56.572308+0000 mon.smithi077 (mon.0) 576 : audit [DBG] from='client.? 172.21.15.77:0/3951473739' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:57.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:57 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:55.500353+0000 mgr.smithi077.amvfyi (mgr.14180) 202 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:57.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:57 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:56.572308+0000 mon.smithi077 (mon.0) 576 : audit [DBG] from='client.? 172.21.15.77:0/3951473739' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:02:57.933 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:02:59.297 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:02:59.321 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:02:59 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:57.501039+0000 mgr.smithi077.amvfyi (mgr.14180) 203 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:59.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:02:59 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:57.501039+0000 mgr.smithi077.amvfyi (mgr.14180) 203 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:02:59.674 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:03:00.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:02:59.304557+0000 mon.smithi077 (mon.0) 577 : audit [DBG] from='client.? 172.21.15.77:0/4113310110' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:00.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:02:59.304557+0000 mon.smithi077 (mon.0) 577 : audit [DBG] from='client.? 172.21.15.77:0/4113310110' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:00.675 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:03:01.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:01 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:02:59.501644+0000 mgr.smithi077.amvfyi (mgr.14180) 204 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:01.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:01 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:02:59.501644+0000 mgr.smithi077.amvfyi (mgr.14180) 204 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:02.127 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:03:02.490 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:03:03.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:03 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:03:01.502090+0000 mgr.smithi077.amvfyi (mgr.14180) 205 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:03.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:03 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:02.134682+0000 mon.smithi077 (mon.0) 578 : audit [DBG] from='client.? 172.21.15.77:0/1194321225' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:03.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:03 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:03:01.502090+0000 mgr.smithi077.amvfyi (mgr.14180) 205 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:03.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:03 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:02.134682+0000 mon.smithi077 (mon.0) 578 : audit [DBG] from='client.? 172.21.15.77:0/1194321225' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:03.491 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:03:04.776 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:03:05.075 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:05 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:03:03.502526+0000 mgr.smithi077.amvfyi (mgr.14180) 206 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:05.075 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:05 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:04.783949+0000 mon.smithi077 (mon.0) 579 : audit [DBG] from='client.? 172.21.15.77:0/3266848723' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:05.132 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:03:05.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:05 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:03:03.502526+0000 mgr.smithi077.amvfyi (mgr.14180) 206 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:05.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:05 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:04.783949+0000 mon.smithi077 (mon.0) 579 : audit [DBG] from='client.? 172.21.15.77:0/3266848723' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:06.133 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:03:07.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:07 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:03:05.502986+0000 mgr.smithi077.amvfyi (mgr.14180) 207 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:07.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:07 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:03:05.502986+0000 mgr.smithi077.amvfyi (mgr.14180) 207 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:07.443 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:03:07.806 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:03:08.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:08 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:07.451143+0000 mon.smithi077 (mon.0) 580 : audit [DBG] from='client.? 172.21.15.77:0/3922376002' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:08.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:08 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:07.451143+0000 mon.smithi077 (mon.0) 580 : audit [DBG] from='client.? 172.21.15.77:0/3922376002' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:08.807 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:03:09.183 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:09 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:03:07.503386+0000 mgr.smithi077.amvfyi (mgr.14180) 208 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:09.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:09 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:03:07.503386+0000 mgr.smithi077.amvfyi (mgr.14180) 208 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:10.164 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:03:10.564 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:03:11.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:11 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:03:09.504032+0000 mgr.smithi077.amvfyi (mgr.14180) 209 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:11.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:11 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:10.172553+0000 mon.smithi077 (mon.0) 581 : audit [DBG] from='client.? 172.21.15.77:0/377357422' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:11.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:11 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:03:09.504032+0000 mgr.smithi077.amvfyi (mgr.14180) 209 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:11.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:11 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:10.172553+0000 mon.smithi077 (mon.0) 581 : audit [DBG] from='client.? 172.21.15.77:0/377357422' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:11.565 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:03:12.869 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:03:13.154 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:13 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:03:11.504480+0000 mgr.smithi077.amvfyi (mgr.14180) 210 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:13.154 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:13 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:12.877112+0000 mon.smithi077 (mon.0) 582 : audit [DBG] from='client.? 172.21.15.77:0/2389506030' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:13.239 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:03:13.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:13 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:03:11.504480+0000 mgr.smithi077.amvfyi (mgr.14180) 210 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:13.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:13 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:12.877112+0000 mon.smithi077 (mon.0) 582 : audit [DBG] from='client.? 172.21.15.77:0/2389506030' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:14.240 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:03:15.259 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:15 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:03:13.504866+0000 mgr.smithi077.amvfyi (mgr.14180) 211 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:15.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:15 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:03:13.504866+0000 mgr.smithi077.amvfyi (mgr.14180) 211 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:15.592 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:03:15.963 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:03:16.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:16 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:15.600225+0000 mon.smithi077 (mon.0) 583 : audit [DBG] from='client.? 172.21.15.77:0/665435092' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:16.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:16 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:15.600225+0000 mon.smithi077 (mon.0) 583 : audit [DBG] from='client.? 172.21.15.77:0/665435092' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:16.964 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:03:17.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:17 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:03:15.505419+0000 mgr.smithi077.amvfyi (mgr.14180) 212 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:17.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:17 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:03:15.505419+0000 mgr.smithi077.amvfyi (mgr.14180) 212 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:18.335 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:03:18.737 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:03:19.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:19 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:03:17.505839+0000 mgr.smithi077.amvfyi (mgr.14180) 213 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:19.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:19 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:18.343285+0000 mon.smithi077 (mon.0) 584 : audit [DBG] from='client.? 172.21.15.77:0/2479021991' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:19.575 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:19 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:03:17.505839+0000 mgr.smithi077.amvfyi (mgr.14180) 213 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:19.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:19 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:18.343285+0000 mon.smithi077 (mon.0) 584 : audit [DBG] from='client.? 172.21.15.77:0/2479021991' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:19.739 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:03:21.076 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:03:21.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:03:19.506384+0000 mgr.smithi077.amvfyi (mgr.14180) 214 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:21.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:21 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:21.083088+0000 mon.smithi077 (mon.0) 585 : audit [DBG] from='client.? 172.21.15.77:0/3826234785' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:21.421 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:03:19.506384+0000 mgr.smithi077.amvfyi (mgr.14180) 214 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:21.421 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:21 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:21.083088+0000 mon.smithi077 (mon.0) 585 : audit [DBG] from='client.? 172.21.15.77:0/3826234785' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:21.464 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:03:22.465 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:03:23.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:23 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:03:21.506816+0000 mgr.smithi077.amvfyi (mgr.14180) 215 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:23.443 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:23 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:03:21.506816+0000 mgr.smithi077.amvfyi (mgr.14180) 215 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:23.785 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:03:24.161 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:03:24.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:24 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:23.793130+0000 mon.smithi077 (mon.0) 586 : audit [DBG] from='client.? 172.21.15.77:0/81729434' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:24.575 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:24 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:23.793130+0000 mon.smithi077 (mon.0) 586 : audit [DBG] from='client.? 172.21.15.77:0/81729434' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:25.163 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:03:25.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:25 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:03:23.507180+0000 mgr.smithi077.amvfyi (mgr.14180) 216 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:25.538 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:25 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:03:23.507180+0000 mgr.smithi077.amvfyi (mgr.14180) 216 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:26.526 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:03:26.928 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:03:27.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:27 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:03:25.507769+0000 mgr.smithi077.amvfyi (mgr.14180) 217 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:27.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:27 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:26.533830+0000 mon.smithi077 (mon.0) 587 : audit [DBG] from='client.? 172.21.15.77:0/1335265917' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:27.575 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:27 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:03:25.507769+0000 mgr.smithi077.amvfyi (mgr.14180) 217 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:27.575 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:27 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:26.533830+0000 mon.smithi077 (mon.0) 587 : audit [DBG] from='client.? 172.21.15.77:0/1335265917' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:27.930 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:03:29.275 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:03:29.301 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:29 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:03:27.508289+0000 mgr.smithi077.amvfyi (mgr.14180) 218 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:29.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:29 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:03:27.508289+0000 mgr.smithi077.amvfyi (mgr.14180) 218 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:29.610 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:03:30.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:30 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:29.283004+0000 mon.smithi077 (mon.0) 588 : audit [DBG] from='client.? 172.21.15.77:0/1646095600' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:30.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:30 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:29.283004+0000 mon.smithi077 (mon.0) 588 : audit [DBG] from='client.? 172.21.15.77:0/1646095600' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:30.611 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:03:31.264 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:31 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:03:29.508664+0000 mgr.smithi077.amvfyi (mgr.14180) 219 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:31.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:31 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:03:29.508664+0000 mgr.smithi077.amvfyi (mgr.14180) 219 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:31.966 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:03:32.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:32 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:31.973825+0000 mon.smithi077 (mon.0) 589 : audit [DBG] from='client.? 172.21.15.77:0/1961693424' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:32.344 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:03:32.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:32 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:31.973825+0000 mon.smithi077 (mon.0) 589 : audit [DBG] from='client.? 172.21.15.77:0/1961693424' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:33.345 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:03:33.575 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:33 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:03:31.509043+0000 mgr.smithi077.amvfyi (mgr.14180) 220 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:33.628 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:33 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:03:31.509043+0000 mgr.smithi077.amvfyi (mgr.14180) 220 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:34.733 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:03:35.119 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:03:35.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:35 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:03:33.509605+0000 mgr.smithi077.amvfyi (mgr.14180) 221 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:35.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:35 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:34.741418+0000 mon.smithi077 (mon.0) 590 : audit [DBG] from='client.? 172.21.15.77:0/2703487890' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:35.628 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:35 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:03:33.509605+0000 mgr.smithi077.amvfyi (mgr.14180) 221 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:35.628 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:35 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:34.741418+0000 mon.smithi077 (mon.0) 590 : audit [DBG] from='client.? 172.21.15.77:0/2703487890' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:36.121 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:03:36.448 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:36 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:35.496703+0000 mon.smithi077 (mon.0) 591 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T09:03:36.448 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:36 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:35.783163+0000 mon.smithi077 (mon.0) 592 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:03:36.448 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:36 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:35.783779+0000 mon.smithi077 (mon.0) 593 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T09:03:36.449 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:36 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:35.787798+0000 mon.smithi077 (mon.0) 594 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T09:03:36.449 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:36 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:35.791167+0000 mon.smithi077 (mon.0) 595 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T09:03:36.449 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:36 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:35.794180+0000 mon.smithi077 (mon.0) 596 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:03:36.449 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:36 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:35.795162+0000 mon.smithi077 (mon.0) 597 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:03:36.449 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:36 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:35.795367+0000 mon.smithi077 (mon.0) 598 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:03:36.449 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:36 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:35.795798+0000 mon.smithi077 (mon.0) 599 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:03:36.464 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:36 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:35.496703+0000 mon.smithi077 (mon.0) 591 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T09:03:36.464 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:36 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:35.783163+0000 mon.smithi077 (mon.0) 592 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:03:36.464 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:36 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:35.783779+0000 mon.smithi077 (mon.0) 593 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T09:03:36.464 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:36 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:35.787798+0000 mon.smithi077 (mon.0) 594 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T09:03:36.464 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:36 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:35.791167+0000 mon.smithi077 (mon.0) 595 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T09:03:36.464 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:36 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:35.794180+0000 mon.smithi077 (mon.0) 596 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:03:36.465 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:36 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:35.795162+0000 mon.smithi077 (mon.0) 597 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T09:03:36.465 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:36 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:35.795367+0000 mon.smithi077 (mon.0) 598 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:03:36.465 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:36 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:35.795798+0000 mon.smithi077 (mon.0) 599 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T09:03:37.273 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:03:35.510124+0000 mgr.smithi077.amvfyi (mgr.14180) 222 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:37.273 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:03:35.788094+0000 mgr.smithi077.amvfyi (mgr.14180) 223 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:37.273 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:03:36.151294+0000 mon.smithi077 (mon.0) 600 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T09:03:37.274 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:37 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:03:36.151328+0000 mon.smithi077 (mon.0) 601 : cluster [INF] Cluster is now healthy 2024-01-13T09:03:37.628 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:03:35.510124+0000 mgr.smithi077.amvfyi (mgr.14180) 222 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:37.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:03:35.788094+0000 mgr.smithi077.amvfyi (mgr.14180) 223 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:37.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:03:36.151294+0000 mon.smithi077 (mon.0) 600 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T09:03:37.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:37 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:03:36.151328+0000 mon.smithi077 (mon.0) 601 : cluster [INF] Cluster is now healthy 2024-01-13T09:03:37.965 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:03:38.301 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:38 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:37.876522+0000 mon.smithi141 (mon.1) 15 : audit [INF] from='client.? 172.21.15.141:0/63737478' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea1790ce-3494-4520-bff4-f7ad531e1117"}]: dispatch 2024-01-13T09:03:38.301 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:38 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:37.876922+0000 mon.smithi077 (mon.0) 602 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea1790ce-3494-4520-bff4-f7ad531e1117"}]: dispatch 2024-01-13T09:03:38.301 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:38 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:37.881267+0000 mon.smithi077 (mon.0) 603 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ea1790ce-3494-4520-bff4-f7ad531e1117"}]': finished 2024-01-13T09:03:38.301 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:38 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:03:37.881348+0000 mon.smithi077 (mon.0) 604 : cluster [DBG] osdmap e30: 1 total, 0 up, 1 in 2024-01-13T09:03:38.301 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:38 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:37.881468+0000 mon.smithi077 (mon.0) 605 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:03:38.301 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:38 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:37.973306+0000 mon.smithi077 (mon.0) 606 : audit [DBG] from='client.? 172.21.15.77:0/2934306474' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:38.353 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":30,"num_osds":1,"num_up_osds":0,"osd_up_since":0,"num_in_osds":1,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:03:38.431 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:38 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:37.876522+0000 mon.smithi141 (mon.1) 15 : audit [INF] from='client.? 172.21.15.141:0/63737478' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea1790ce-3494-4520-bff4-f7ad531e1117"}]: dispatch 2024-01-13T09:03:38.431 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:38 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:37.876922+0000 mon.smithi077 (mon.0) 602 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea1790ce-3494-4520-bff4-f7ad531e1117"}]: dispatch 2024-01-13T09:03:38.431 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:38 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:37.881267+0000 mon.smithi077 (mon.0) 603 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ea1790ce-3494-4520-bff4-f7ad531e1117"}]': finished 2024-01-13T09:03:38.431 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:38 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:03:37.881348+0000 mon.smithi077 (mon.0) 604 : cluster [DBG] osdmap e30: 1 total, 0 up, 1 in 2024-01-13T09:03:38.431 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:38 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:37.881468+0000 mon.smithi077 (mon.0) 605 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:03:38.431 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:38 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:37.973306+0000 mon.smithi077 (mon.0) 606 : audit [DBG] from='client.? 172.21.15.77:0/2934306474' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:39.355 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:03:39.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:39 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:03:37.788295+0000 mgr.smithi077.amvfyi (mgr.14180) 224 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:39.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:39 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:38.166821+0000 mon.smithi077 (mon.0) 607 : audit [INF] from='client.? 172.21.15.77:0/2479015240' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f1480d0-0260-4e80-9c3e-47ed07987ef2"}]: dispatch 2024-01-13T09:03:39.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:39 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:38.169064+0000 mon.smithi141 (mon.1) 16 : audit [INF] from='client.? 172.21.15.141:0/731376540' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:03:39.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:39 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:38.174357+0000 mon.smithi077 (mon.0) 608 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:03:39.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:39 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:38.178708+0000 mon.smithi077 (mon.0) 609 : audit [INF] from='client.? 172.21.15.77:0/2479015240' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8f1480d0-0260-4e80-9c3e-47ed07987ef2"}]': finished 2024-01-13T09:03:39.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:39 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:03:38.179423+0000 mon.smithi077 (mon.0) 610 : cluster [DBG] osdmap e31: 2 total, 0 up, 2 in 2024-01-13T09:03:39.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:39 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:38.179508+0000 mon.smithi077 (mon.0) 611 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:03:39.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:39 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:38.179741+0000 mon.smithi077 (mon.0) 612 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-13T09:03:39.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:39 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:38.184819+0000 mon.smithi077 (mon.0) 613 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:03:39.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:39 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:03:38.184867+0000 mon.smithi077 (mon.0) 614 : cluster [DBG] osdmap e32: 1 total, 0 up, 1 in 2024-01-13T09:03:39.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:39 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:38.184995+0000 mon.smithi077 (mon.0) 615 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-13T09:03:39.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:39 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:38.480541+0000 mon.smithi077 (mon.0) 616 : audit [INF] from='client.? 172.21.15.77:0/87898002' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:03:39.577 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:39 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:38.484244+0000 mon.smithi077 (mon.0) 617 : audit [INF] from='client.? 172.21.15.77:0/87898002' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:03:39.577 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:39 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:03:38.484282+0000 mon.smithi077 (mon.0) 618 : cluster [DBG] osdmap e33: 0 total, 0 up, 0 in 2024-01-13T09:03:39.577 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:39 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:38.542648+0000 mon.smithi077 (mon.0) 619 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T09:03:39.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:39 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:03:37.788295+0000 mgr.smithi077.amvfyi (mgr.14180) 224 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:39.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:39 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:38.166821+0000 mon.smithi077 (mon.0) 607 : audit [INF] from='client.? 172.21.15.77:0/2479015240' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f1480d0-0260-4e80-9c3e-47ed07987ef2"}]: dispatch 2024-01-13T09:03:39.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:39 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:38.169064+0000 mon.smithi141 (mon.1) 16 : audit [INF] from='client.? 172.21.15.141:0/731376540' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:03:39.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:39 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:38.174357+0000 mon.smithi077 (mon.0) 608 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:03:39.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:39 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:38.178708+0000 mon.smithi077 (mon.0) 609 : audit [INF] from='client.? 172.21.15.77:0/2479015240' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8f1480d0-0260-4e80-9c3e-47ed07987ef2"}]': finished 2024-01-13T09:03:39.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:39 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:03:38.179423+0000 mon.smithi077 (mon.0) 610 : cluster [DBG] osdmap e31: 2 total, 0 up, 2 in 2024-01-13T09:03:39.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:39 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:38.179508+0000 mon.smithi077 (mon.0) 611 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T09:03:39.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:39 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:38.179741+0000 mon.smithi077 (mon.0) 612 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-13T09:03:39.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:39 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:38.184819+0000 mon.smithi077 (mon.0) 613 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:03:39.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:39 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:03:38.184867+0000 mon.smithi077 (mon.0) 614 : cluster [DBG] osdmap e32: 1 total, 0 up, 1 in 2024-01-13T09:03:39.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:39 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:38.184995+0000 mon.smithi077 (mon.0) 615 : audit [DBG] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-13T09:03:39.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:39 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:38.480541+0000 mon.smithi077 (mon.0) 616 : audit [INF] from='client.? 172.21.15.77:0/87898002' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T09:03:39.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:39 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:38.484244+0000 mon.smithi077 (mon.0) 617 : audit [INF] from='client.? 172.21.15.77:0/87898002' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]': finished 2024-01-13T09:03:39.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:39 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:03:38.484282+0000 mon.smithi077 (mon.0) 618 : cluster [DBG] osdmap e33: 0 total, 0 up, 0 in 2024-01-13T09:03:39.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:39 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:38.542648+0000 mon.smithi077 (mon.0) 619 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T09:03:40.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T09:03:38.650593+0000 mgr.smithi077.amvfyi (mgr.14180) 225 : cephadm [ERR] executing create_from_spec_one(([('smithi077', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi141', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:03:40.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T09:03:40.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:03:40.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return f(*arg) 2024-01-13T09:03:40.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:03:40.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:03:40.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:03:40.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: code, '\n'.join(err))) 2024-01-13T09:03:40.326 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T09:03:40.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp9ppqwdgu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpg371w_gd:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:03:40.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:03:40.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:03:40.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:03:40.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new ea1790ce-3494-4520-bff4-f7ad531e1117 2024-01-13T09:03:40.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:03:40.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:03:40.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:03:40.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:03:40.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:03:40.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.prepare() 2024-01-13T09:03:40.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:03:40.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:03:40.327 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:03:40.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:03:40.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:03:40.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:03:40.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:03:40.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:03:40.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:03:40.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T09:03:40.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:03:40.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T09:03:40.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:03:40.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:03:40.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:03:40.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:03:40.328 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:03:40.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:03:40.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:03:40.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:03:40.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:03:40.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:03:40.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T09:03:40.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:03:40.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:03:40.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:03:40.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T09:03:40.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:03:40.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:03:40.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:03:40.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:03:40.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:03:40.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:03:40.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:03:40.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:03:40.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:03:40.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:03:40.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:03:40.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:03:40.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:03:40.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:03:40.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:03:40.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:03:40.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:03:40.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:03:40.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:03:40.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:03:40.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:03:40.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:03:40.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:03:40.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T09:03:40.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:03:40.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: main() 2024-01-13T09:03:40.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:03:40.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: r = ctx.func(ctx) 2024-01-13T09:03:40.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:03:40.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:03:40.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:03:40.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:03:40.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:03:40.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:03:40.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:03:40.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:03:40.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:03:40.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:03:40.334 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:03:40.335 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:03:40.335 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp9ppqwdgu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpg371w_gd:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:03:40.335 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T09:03:38.944445+0000 mgr.smithi077.amvfyi (mgr.14180) 226 : cephadm [ERR] executing create_from_spec_one(([('smithi077', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi141', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:03:40.335 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T09:03:40.335 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:03:40.335 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return f(*arg) 2024-01-13T09:03:40.335 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:03:40.335 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:03:40.335 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:03:40.335 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: code, '\n'.join(err))) 2024-01-13T09:03:40.335 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi077/config 2024-01-13T09:03:40.335 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnbayw1kp:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp_t9mxlp4:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:03:40.335 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:03:40.335 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:03:40.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:03:40.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 8f1480d0-0260-4e80-9c3e-47ed07987ef2 2024-01-13T09:03:40.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:03:40.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 --yes-i-really-mean-it 2024-01-13T09:03:40.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr stderr: purged osd.1 2024-01-13T09:03:40.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:03:40.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:03:40.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.prepare() 2024-01-13T09:03:40.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:03:40.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:03:40.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:03:40.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:03:40.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:03:40.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:03:40.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:03:40.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:03:40.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:03:40.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T09:03:40.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:03:40.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T09:03:40.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:03:40.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:03:40.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:03:40.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:03:40.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:03:40.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:03:40.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:03:40.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:03:40.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:03:40.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:03:40.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T09:03:40.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:03:40.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:03:40.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:03:40.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T09:03:40.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:03:40.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:03:40.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:03:40.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:03:40.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:03:40.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:03:40.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:03:40.338 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:03:40.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:03:40.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:03:40.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:03:40.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:03:40.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:03:40.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:03:40.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:03:40.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:03:40.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:03:40.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:03:40.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:03:40.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:03:40.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:03:40.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:03:40.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 1 2024-01-13T09:03:40.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T09:03:40.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:03:40.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: main() 2024-01-13T09:03:40.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:03:40.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: r = ctx.func(ctx) 2024-01-13T09:03:40.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:03:40.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:03:40.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:03:40.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:03:40.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:03:40.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:03:40.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:03:40.341 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:03:40.341 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:03:40.341 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:03:40.341 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:03:40.341 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:03:40.341 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnbayw1kp:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp_t9mxlp4:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:03:40.341 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cephadm 2024-01-13T09:03:38.945581+0000 mgr.smithi077.amvfyi (mgr.14180) 227 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T09:03:40.341 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: service_id: all-available-devices 2024-01-13T09:03:40.341 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: service_name: osd.all-available-devices 2024-01-13T09:03:40.341 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: placement: 2024-01-13T09:03:40.341 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: host_pattern: '*' 2024-01-13T09:03:40.341 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: spec: 2024-01-13T09:03:40.341 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: data_devices: 2024-01-13T09:03:40.341 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: all: true 2024-01-13T09:03:40.341 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: filter_logic: AND 2024-01-13T09:03:40.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: objectstore: bluestore 2024-01-13T09:03:40.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T09:03:40.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp9ppqwdgu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpg371w_gd:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:03:40.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:03:40.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:03:40.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:03:40.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new ea1790ce-3494-4520-bff4-f7ad531e1117 2024-01-13T09:03:40.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:03:40.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:03:40.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:03:40.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:03:40.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:03:40.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.prepare() 2024-01-13T09:03:40.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:03:40.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:03:40.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:03:40.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:03:40.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:03:40.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:03:40.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:03:40.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:03:40.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:03:40.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T09:03:40.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:03:40.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T09:03:40.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:03:40.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:03:40.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:03:40.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:03:40.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:03:40.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:03:40.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:03:40.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:03:40.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:03:40.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:03:40.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T09:03:40.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:03:40.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:03:40.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:03:40.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T09:03:40.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:03:40.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:03:40.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:03:40.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:03:40.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:03:40.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:03:40.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:03:40.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:03:40.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:03:40.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:03:40.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:03:40.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:03:40.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:03:40.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:03:40.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:03:40.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:03:40.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:03:40.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:03:40.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:03:40.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:03:40.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:03:40.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:03:40.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:03:40.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T09:03:40.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:03:40.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: main() 2024-01-13T09:03:40.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:03:40.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: r = ctx.func(ctx) 2024-01-13T09:03:40.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:03:40.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:03:40.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:03:40.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:03:40.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:03:40.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:03:40.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:03:40.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:03:40.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:03:40.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:03:40.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:03:40.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:03:40.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp9ppqwdgu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpg371w_gd:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:03:40.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T09:03:40.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T09:03:40.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: if self._apply_service(spec): 2024-01-13T09:03:40.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T09:03:40.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T09:03:40.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T09:03:40.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T09:03:40.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T09:03:40.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T09:03:40.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T09:03:40.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T09:03:40.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T09:03:40.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: raise self._value 2024-01-13T09:03:40.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T09:03:40.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: result = (True, func(*args, **kwds)) 2024-01-13T09:03:40.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T09:03:40.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return list(map(*args)) 2024-01-13T09:03:40.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:03:40.348 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return f(*arg) 2024-01-13T09:03:40.349 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:03:40.349 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:03:40.349 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:03:40.349 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: code, '\n'.join(err))) 2024-01-13T09:03:40.349 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T09:03:40.349 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp9ppqwdgu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpg371w_gd:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:03:40.349 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:03:40.349 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:03:40.349 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:03:40.349 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new ea1790ce-3494-4520-bff4-f7ad531e1117 2024-01-13T09:03:40.349 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:03:40.349 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:03:40.349 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:03:40.349 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:03:40.350 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:03:40.350 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.prepare() 2024-01-13T09:03:40.350 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:03:40.350 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:03:40.350 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:03:40.350 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:03:40.350 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:03:40.350 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:03:40.350 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:03:40.350 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:03:40.350 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:03:40.350 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T09:03:40.350 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:03:40.350 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr 2024-01-13T09:03:40.350 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:03:40.351 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:03:40.351 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:03:40.351 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:03:40.351 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:03:40.351 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:03:40.351 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:03:40.351 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:03:40.351 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:03:40.351 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:03:40.351 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T09:03:40.351 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:03:40.351 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:03:40.351 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:03:40.351 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr instance.main() 2024-01-13T09:03:40.352 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:03:40.352 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:03:40.352 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:03:40.352 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:03:40.352 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:03:40.352 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:03:40.352 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:03:40.352 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:03:40.352 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:03:40.352 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:03:40.352 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:03:40.352 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:03:40.352 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:03:40.353 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:03:40.353 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:03:40.353 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:03:40.353 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:03:40.353 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:03:40.353 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:03:40.353 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:03:40.353 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:03:40.353 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:03:40.353 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:03:40.353 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: Traceback (most recent call last): 2024-01-13T09:03:40.353 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:03:40.353 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: main() 2024-01-13T09:03:40.354 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:03:40.354 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: r = ctx.func(ctx) 2024-01-13T09:03:40.354 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:03:40.354 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:03:40.354 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:03:40.354 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:03:40.354 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:03:40.354 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:03:40.354 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:03:40.354 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: return func(ctx) 2024-01-13T09:03:40.354 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:03:40.354 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:03:40.355 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:03:40.355 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:03:40.355 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp9ppqwdgu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpg371w_gd:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:03:40.355 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:03:38.946090+0000 mgr.smithi077.amvfyi (mgr.14180) 228 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:40.355 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:40 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:03:39.541030+0000 mon.smithi077 (mon.0) 620 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T09:03:40.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T09:03:38.650593+0000 mgr.smithi077.amvfyi (mgr.14180) 225 : cephadm [ERR] executing create_from_spec_one(([('smithi077', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi141', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:03:40.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T09:03:40.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:03:40.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return f(*arg) 2024-01-13T09:03:40.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:03:40.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:03:40.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:03:40.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: code, '\n'.join(err))) 2024-01-13T09:03:40.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T09:03:40.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp9ppqwdgu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpg371w_gd:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:03:40.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:03:40.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:03:40.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:03:40.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new ea1790ce-3494-4520-bff4-f7ad531e1117 2024-01-13T09:03:40.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:03:40.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:03:40.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:03:40.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:03:40.630 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:03:40.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.prepare() 2024-01-13T09:03:40.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:03:40.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:03:40.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:03:40.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:03:40.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:03:40.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:03:40.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:03:40.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:03:40.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:03:40.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T09:03:40.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:03:40.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T09:03:40.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:03:40.631 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:03:40.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:03:40.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:03:40.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:03:40.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:03:40.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:03:40.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:03:40.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:03:40.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:03:40.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T09:03:40.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:03:40.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:03:40.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:03:40.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T09:03:40.632 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:03:40.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:03:40.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:03:40.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:03:40.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:03:40.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:03:40.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:03:40.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:03:40.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:03:40.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:03:40.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:03:40.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:03:40.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:03:40.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:03:40.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:03:40.633 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:03:40.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:03:40.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:03:40.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:03:40.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:03:40.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:03:40.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:03:40.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:03:40.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T09:03:40.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:03:40.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: main() 2024-01-13T09:03:40.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:03:40.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: r = ctx.func(ctx) 2024-01-13T09:03:40.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:03:40.634 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:03:40.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:03:40.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:03:40.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:03:40.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:03:40.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:03:40.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:03:40.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:03:40.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:03:40.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:03:40.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:03:40.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp9ppqwdgu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpg371w_gd:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:03:40.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T09:03:38.944445+0000 mgr.smithi077.amvfyi (mgr.14180) 226 : cephadm [ERR] executing create_from_spec_one(([('smithi077', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi141', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T09:03:40.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T09:03:40.635 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:03:40.636 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return f(*arg) 2024-01-13T09:03:40.636 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:03:40.636 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:03:40.636 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:03:40.636 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: code, '\n'.join(err))) 2024-01-13T09:03:40.636 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi077/config 2024-01-13T09:03:40.638 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnbayw1kp:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp_t9mxlp4:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:03:40.638 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:03:40.638 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:03:40.638 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:03:40.638 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 8f1480d0-0260-4e80-9c3e-47ed07987ef2 2024-01-13T09:03:40.638 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:03:40.638 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 --yes-i-really-mean-it 2024-01-13T09:03:40.638 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr stderr: purged osd.1 2024-01-13T09:03:40.638 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:03:40.638 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:03:40.639 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.prepare() 2024-01-13T09:03:40.639 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:03:40.639 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:03:40.639 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:03:40.639 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:03:40.639 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:03:40.639 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:03:40.639 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:03:40.639 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:03:40.639 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:03:40.640 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T09:03:40.640 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:03:40.640 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T09:03:40.640 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:03:40.640 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:03:40.640 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:03:40.640 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:03:40.640 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:03:40.640 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:03:40.640 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:03:40.640 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:03:40.640 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:03:40.640 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:03:40.640 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T09:03:40.641 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:03:40.641 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:03:40.641 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:03:40.641 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T09:03:40.641 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:03:40.641 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:03:40.641 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:03:40.641 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:03:40.641 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:03:40.641 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:03:40.641 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:03:40.641 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:03:40.641 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:03:40.641 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:03:40.641 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:03:40.642 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:03:40.642 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:03:40.642 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:03:40.642 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:03:40.642 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:03:40.642 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:03:40.642 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:03:40.643 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:03:40.643 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:03:40.643 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:03:40.643 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:03:40.643 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 1 2024-01-13T09:03:40.643 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T09:03:40.643 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:03:40.643 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: main() 2024-01-13T09:03:40.643 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:03:40.643 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: r = ctx.func(ctx) 2024-01-13T09:03:40.643 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:03:40.643 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:03:40.643 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:03:40.643 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:03:40.643 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:03:40.644 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:03:40.644 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:03:40.644 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:03:40.644 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:03:40.644 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:03:40.644 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:03:40.644 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:03:40.644 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpnbayw1kp:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp_t9mxlp4:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:03:40.644 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cephadm 2024-01-13T09:03:38.945581+0000 mgr.smithi077.amvfyi (mgr.14180) 227 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T09:03:40.644 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: service_id: all-available-devices 2024-01-13T09:03:40.644 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: service_name: osd.all-available-devices 2024-01-13T09:03:40.644 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: placement: 2024-01-13T09:03:40.644 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: host_pattern: '*' 2024-01-13T09:03:40.645 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: spec: 2024-01-13T09:03:40.645 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: data_devices: 2024-01-13T09:03:40.645 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: all: true 2024-01-13T09:03:40.645 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: filter_logic: AND 2024-01-13T09:03:40.645 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: objectstore: bluestore 2024-01-13T09:03:40.645 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T09:03:40.645 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp9ppqwdgu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpg371w_gd:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:03:40.645 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:03:40.645 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:03:40.645 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:03:40.645 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new ea1790ce-3494-4520-bff4-f7ad531e1117 2024-01-13T09:03:40.645 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:03:40.645 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:03:40.645 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:03:40.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:03:40.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:03:40.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.prepare() 2024-01-13T09:03:40.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:03:40.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:03:40.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:03:40.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:03:40.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:03:40.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:03:40.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:03:40.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:03:40.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:03:40.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T09:03:40.646 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:03:40.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T09:03:40.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:03:40.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:03:40.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:03:40.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:03:40.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:03:40.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:03:40.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:03:40.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:03:40.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:03:40.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:03:40.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T09:03:40.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:03:40.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:03:40.647 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:03:40.648 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T09:03:40.648 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:03:40.648 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:03:40.648 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:03:40.648 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:03:40.648 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:03:40.648 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:03:40.648 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:03:40.648 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:03:40.648 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:03:40.648 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:03:40.648 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:03:40.648 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:03:40.648 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:03:40.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:03:40.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:03:40.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:03:40.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:03:40.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:03:40.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:03:40.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:03:40.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:03:40.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:03:40.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:03:40.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T09:03:40.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:03:40.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: main() 2024-01-13T09:03:40.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:03:40.649 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: r = ctx.func(ctx) 2024-01-13T09:03:40.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:03:40.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:03:40.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:03:40.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:03:40.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:03:40.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:03:40.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:03:40.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:03:40.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:03:40.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:03:40.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:03:40.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:03:40.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp9ppqwdgu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpg371w_gd:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:03:40.650 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T09:03:40.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T09:03:40.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: if self._apply_service(spec): 2024-01-13T09:03:40.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T09:03:40.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T09:03:40.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T09:03:40.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T09:03:40.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T09:03:40.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T09:03:40.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T09:03:40.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T09:03:40.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T09:03:40.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: raise self._value 2024-01-13T09:03:40.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T09:03:40.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: result = (True, func(*args, **kwds)) 2024-01-13T09:03:40.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T09:03:40.651 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return list(map(*args)) 2024-01-13T09:03:40.652 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T09:03:40.652 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return f(*arg) 2024-01-13T09:03:40.652 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T09:03:40.652 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T09:03:40.652 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T09:03:40.652 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: code, '\n'.join(err))) 2024-01-13T09:03:40.652 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/mon.smithi141/config 2024-01-13T09:03:40.652 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp9ppqwdgu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpg371w_gd:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:03:40.653 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T09:03:40.653 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T09:03:40.653 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T09:03:40.653 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new ea1790ce-3494-4520-bff4-f7ad531e1117 2024-01-13T09:03:40.653 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T09:03:40.653 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T09:03:40.653 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T09:03:40.653 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:03:40.653 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T09:03:40.653 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.prepare() 2024-01-13T09:03:40.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:03:40.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:03:40.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T09:03:40.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T09:03:40.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T09:03:40.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T09:03:40.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T09:03:40.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T09:03:40.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T09:03:40.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T09:03:40.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T09:03:40.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr 2024-01-13T09:03:40.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T09:03:40.654 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T09:03:40.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T09:03:40.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T09:03:40.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.main(self.argv) 2024-01-13T09:03:40.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T09:03:40.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T09:03:40.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T09:03:40.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T09:03:40.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:03:40.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T09:03:40.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T09:03:40.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T09:03:40.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T09:03:40.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr instance.main() 2024-01-13T09:03:40.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:03:40.655 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:03:40.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T09:03:40.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self._execute(plan) 2024-01-13T09:03:40.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T09:03:40.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T09:03:40.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:03:40.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:03:40.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T09:03:40.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T09:03:40.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T09:03:40.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T09:03:40.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T09:03:40.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T09:03:40.656 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T09:03:40.657 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr self.zap_osd() 2024-01-13T09:03:40.657 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T09:03:40.657 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T09:03:40.657 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T09:03:40.657 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T09:03:40.657 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T09:03:40.657 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T09:03:40.657 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T09:03:40.657 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: Traceback (most recent call last): 2024-01-13T09:03:40.657 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T09:03:40.657 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: main() 2024-01-13T09:03:40.657 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T09:03:40.657 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: r = ctx.func(ctx) 2024-01-13T09:03:40.657 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T09:03:40.658 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:03:40.658 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T09:03:40.658 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:03:40.658 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T09:03:40.658 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:03:40.658 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T09:03:40.658 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: return func(ctx) 2024-01-13T09:03:40.658 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T09:03:40.658 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T09:03:40.658 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: File "/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T09:03:40.658 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T09:03:40.658 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi141 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp9ppqwdgu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpg371w_gd:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T09:03:40.658 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:03:38.946090+0000 mgr.smithi077.amvfyi (mgr.14180) 228 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:40.658 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:40 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:03:39.541030+0000 mon.smithi077 (mon.0) 620 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T09:03:40.747 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:03:41.118 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:03:41.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:41 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:40.755045+0000 mon.smithi077 (mon.0) 621 : audit [DBG] from='client.? 172.21.15.77:0/2374480478' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:41.628 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:41 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:40.755045+0000 mon.smithi077 (mon.0) 621 : audit [DBG] from='client.? 172.21.15.77:0/2374480478' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:42.119 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:03:42.479 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:42 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:03:40.946554+0000 mgr.smithi077.amvfyi (mgr.14180) 229 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:42.628 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:42 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:03:40.946554+0000 mgr.smithi077.amvfyi (mgr.14180) 229 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:43.489 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:03:43.860 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:03:44.575 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:44 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:03:42.947004+0000 mgr.smithi077.amvfyi (mgr.14180) 230 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:44.575 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:44 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:43.496991+0000 mon.smithi077 (mon.0) 622 : audit [DBG] from='client.? 172.21.15.77:0/2166345427' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:44.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:44 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:03:42.947004+0000 mgr.smithi077.amvfyi (mgr.14180) 230 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:44.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:44 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:43.496991+0000 mon.smithi077 (mon.0) 622 : audit [DBG] from='client.? 172.21.15.77:0/2166345427' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:44.861 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:03:46.240 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:03:46.533 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:46 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:03:44.947421+0000 mgr.smithi077.amvfyi (mgr.14180) 231 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:46.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:46 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:03:44.947421+0000 mgr.smithi077.amvfyi (mgr.14180) 231 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:46.634 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:03:47.575 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:47 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:46.248430+0000 mon.smithi077 (mon.0) 623 : audit [DBG] from='client.? 172.21.15.77:0/3519131822' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:47.628 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:47 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:46.248430+0000 mon.smithi077 (mon.0) 623 : audit [DBG] from='client.? 172.21.15.77:0/3519131822' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:47.635 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:03:48.271 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:48 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:03:46.948094+0000 mgr.smithi077.amvfyi (mgr.14180) 232 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:48.628 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:48 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:03:46.948094+0000 mgr.smithi077.amvfyi (mgr.14180) 232 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:48.965 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:03:49.298 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:49 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:48.973109+0000 mon.smithi077 (mon.0) 624 : audit [DBG] from='client.? 172.21.15.77:0/4216914780' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:49.333 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:03:49.628 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:49 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:48.973109+0000 mon.smithi077 (mon.0) 624 : audit [DBG] from='client.? 172.21.15.77:0/4216914780' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:50.335 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:03:50.575 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:50 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:03:48.948486+0000 mgr.smithi077.amvfyi (mgr.14180) 233 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:50.628 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:50 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:03:48.948486+0000 mgr.smithi077.amvfyi (mgr.14180) 233 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:51.445 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:51 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:03:50.949041+0000 mgr.smithi077.amvfyi (mgr.14180) 234 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:51.628 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:51 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:03:50.949041+0000 mgr.smithi077.amvfyi (mgr.14180) 234 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:51.781 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:03:52.159 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:03:52.575 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:52 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:51.789884+0000 mon.smithi077 (mon.0) 625 : audit [DBG] from='client.? 172.21.15.77:0/2794572719' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:52.629 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:52 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:51.789884+0000 mon.smithi077 (mon.0) 625 : audit [DBG] from='client.? 172.21.15.77:0/2794572719' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:53.161 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:03:53.503 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:53 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:03:52.949670+0000 mgr.smithi077.amvfyi (mgr.14180) 235 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:53.628 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:53 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:03:52.949670+0000 mgr.smithi077.amvfyi (mgr.14180) 235 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:54.552 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:03:54.825 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:54 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:53.542581+0000 mon.smithi077 (mon.0) 626 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T09:03:54.878 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:54 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:53.542581+0000 mon.smithi077 (mon.0) 626 : audit [INF] from='mgr.14180 172.21.15.77:0/2066918987' entity='mgr.smithi077.amvfyi' 2024-01-13T09:03:54.934 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:03:55.825 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:55 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:54.559866+0000 mon.smithi077 (mon.0) 627 : audit [DBG] from='client.? 172.21.15.77:0/3203178159' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:55.825 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:55 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:03:54.950081+0000 mgr.smithi077.amvfyi (mgr.14180) 236 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:55.878 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:55 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:54.559866+0000 mon.smithi077 (mon.0) 627 : audit [DBG] from='client.? 172.21.15.77:0/3203178159' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:55.879 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:55 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:03:54.950081+0000 mgr.smithi077.amvfyi (mgr.14180) 236 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:55.934 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:03:57.223 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:03:57.601 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:03:58.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:58 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:03:56.950651+0000 mgr.smithi077.amvfyi (mgr.14180) 237 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:58.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:03:58 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:57.231077+0000 mon.smithi077 (mon.0) 628 : audit [DBG] from='client.? 172.21.15.77:0/232158001' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:58.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:58 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:03:56.950651+0000 mgr.smithi077.amvfyi (mgr.14180) 237 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:03:58.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:03:58 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:57.231077+0000 mon.smithi077 (mon.0) 628 : audit [DBG] from='client.? 172.21.15.77:0/232158001' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:03:58.602 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:03:59.953 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:04:00.153 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:04:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:03:58.951059+0000 mgr.smithi077.amvfyi (mgr.14180) 238 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:04:00.153 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:04:00 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:03:59.961701+0000 mon.smithi077 (mon.0) 629 : audit [DBG] from='client.? 172.21.15.77:0/901905368' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:04:00.358 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:04:00.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:04:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:03:58.951059+0000 mgr.smithi077.amvfyi (mgr.14180) 238 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:04:00.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:04:00 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:03:59.961701+0000 mon.smithi077 (mon.0) 629 : audit [DBG] from='client.? 172.21.15.77:0/901905368' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:04:01.359 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:04:02.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:04:02 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:04:00.951492+0000 mgr.smithi077.amvfyi (mgr.14180) 239 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:04:02.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:04:02 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:04:00.951492+0000 mgr.smithi077.amvfyi (mgr.14180) 239 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:04:02.721 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:04:03.090 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:04:03.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:04:03 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:04:02.729555+0000 mon.smithi077 (mon.0) 630 : audit [DBG] from='client.? 172.21.15.77:0/1315819987' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:04:03.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:04:03 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:04:02.729555+0000 mon.smithi077 (mon.0) 630 : audit [DBG] from='client.? 172.21.15.77:0/1315819987' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:04:04.091 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:04:04.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:04:04 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:04:02.951862+0000 mgr.smithi077.amvfyi (mgr.14180) 240 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:04:04.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:04:04 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:04:02.951862+0000 mgr.smithi077.amvfyi (mgr.14180) 240 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:04:05.394 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:04:05.759 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:04:06.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:04:06 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:04:04.952283+0000 mgr.smithi077.amvfyi (mgr.14180) 241 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:04:06.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:04:06 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:04:05.402965+0000 mon.smithi077 (mon.0) 631 : audit [DBG] from='client.? 172.21.15.77:0/1703877782' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:04:06.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:04:06 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:04:04.952283+0000 mgr.smithi077.amvfyi (mgr.14180) 241 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:04:06.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:04:06 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:04:05.402965+0000 mon.smithi077 (mon.0) 631 : audit [DBG] from='client.? 172.21.15.77:0/1703877782' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:04:06.760 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:04:08.098 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:04:08.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:04:08 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:04:06.952998+0000 mgr.smithi077.amvfyi (mgr.14180) 242 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:04:08.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:04:08 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:04:06.952998+0000 mgr.smithi077.amvfyi (mgr.14180) 242 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:04:08.465 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:04:09.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:04:09 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:04:08.107072+0000 mon.smithi077 (mon.0) 632 : audit [DBG] from='client.? 172.21.15.77:0/2856166903' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:04:09.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:04:09 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:04:08.107072+0000 mon.smithi077 (mon.0) 632 : audit [DBG] from='client.? 172.21.15.77:0/2856166903' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:04:09.466 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:04:10.135 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:04:10 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:04:08.953459+0000 mgr.smithi077.amvfyi (mgr.14180) 243 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:04:10.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:04:10 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:04:08.953459+0000 mgr.smithi077.amvfyi (mgr.14180) 243 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:04:10.853 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:04:11.163 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:04:11 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:04:10.861221+0000 mon.smithi077 (mon.0) 633 : audit [DBG] from='client.? 172.21.15.77:0/228711362' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:04:11.231 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:04:11.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:04:11 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:04:10.861221+0000 mon.smithi077 (mon.0) 633 : audit [DBG] from='client.? 172.21.15.77:0/228711362' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:04:12.232 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:04:12.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:04:12 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:04:10.953994+0000 mgr.smithi077.amvfyi (mgr.14180) 244 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:04:12.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:04:12 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:04:10.953994+0000 mgr.smithi077.amvfyi (mgr.14180) 244 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:04:13.553 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:04:13.939 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:04:14.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:04:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:04:12.954749+0000 mgr.smithi077.amvfyi (mgr.14180) 245 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:04:14.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:04:14 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:04:13.561081+0000 mon.smithi077 (mon.0) 634 : audit [DBG] from='client.? 172.21.15.77:0/1227741587' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:04:14.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:04:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:04:12.954749+0000 mgr.smithi077.amvfyi (mgr.14180) 245 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:04:14.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:04:14 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:04:13.561081+0000 mon.smithi077 (mon.0) 634 : audit [DBG] from='client.? 172.21.15.77:0/1227741587' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:04:14.940 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:04:16.312 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:04:16.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:04:16 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:04:14.955150+0000 mgr.smithi077.amvfyi (mgr.14180) 246 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:04:16.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:04:16 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:04:14.955150+0000 mgr.smithi077.amvfyi (mgr.14180) 246 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:04:16.722 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:04:17.379 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:04:17 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:04:16.320505+0000 mon.smithi077 (mon.0) 635 : audit [DBG] from='client.? 172.21.15.77:0/597420988' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:04:17.575 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:04:17 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:04:16.320505+0000 mon.smithi077 (mon.0) 635 : audit [DBG] from='client.? 172.21.15.77:0/597420988' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:04:17.724 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:04:18.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:04:18 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:04:16.955627+0000 mgr.smithi077.amvfyi (mgr.14180) 247 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:04:18.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:04:18 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:04:16.955627+0000 mgr.smithi077.amvfyi (mgr.14180) 247 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:04:19.067 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:04:19.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:04:19 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:04:19.075165+0000 mon.smithi077 (mon.0) 636 : audit [DBG] from='client.? 172.21.15.77:0/2678385652' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:04:19.420 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:04:19 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:04:19.075165+0000 mon.smithi077 (mon.0) 636 : audit [DBG] from='client.? 172.21.15.77:0/2678385652' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:04:19.447 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:04:20.325 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:04:20 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:04:18.956049+0000 mgr.smithi077.amvfyi (mgr.14180) 248 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:04:20.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:04:20 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:04:18.956049+0000 mgr.smithi077.amvfyi (mgr.14180) 248 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:04:20.448 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T09:04:21.852 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-01-13T09:04:22.237 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:04:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: cluster 2024-01-13T09:04:20.956602+0000 mgr.smithi077.amvfyi (mgr.14180) 249 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:04:22.237 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:04:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: audit 2024-01-13T09:04:21.860419+0000 mon.smithi077 (mon.0) 637 : audit [DBG] from='client.? 172.21.15.77:0/4115506953' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:04:22.256 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705136353,"num_remapped_pgs":0} 2024-01-13T09:04:22.256 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 779, in ceph_osds while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (121) after waiting for 120 seconds 2024-01-13T09:04:22.257 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-13T09:04:22.288 DEBUG:teuthology.orchestra.run.smithi141:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-13T09:04:22.342 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-01-13T09:04:22.342 DEBUG:teuthology.orchestra.run.smithi077:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-13T09:04:22.359 DEBUG:teuthology.orchestra.run.smithi141:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-13T09:04:22.377 INFO:tasks.cephadm:Stopping all daemons... 2024-01-13T09:04:22.377 INFO:tasks.cephadm.mon.smithi077:Stopping mon.smithi077... 2024-01-13T09:04:22.378 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-99e41a38-b1f1-11ee-95ac-87774f69a715@mon.smithi077 2024-01-13T09:04:22.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:04:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: cluster 2024-01-13T09:04:20.956602+0000 mgr.smithi077.amvfyi (mgr.14180) 249 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T09:04:22.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:04:22 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: audit 2024-01-13T09:04:21.860419+0000 mon.smithi077 (mon.0) 637 : audit [DBG] from='client.? 172.21.15.77:0/4115506953' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T09:04:22.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:04:22 smithi077 systemd[1]: Stopping Ceph mon.smithi077 for 99e41a38-b1f1-11ee-95ac-87774f69a715... 2024-01-13T09:04:22.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:04:22 smithi077 bash[70797]: Error: no container with name or ID "ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon.smithi077" found: no such container 2024-01-13T09:04:22.951 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:04:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T09:04:22.636+0000 7f57b1252700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi077 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-13T09:04:22.951 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:04:22 smithi077 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077[30856]: debug 2024-01-13T09:04:22.636+0000 7f57b1252700 -1 mon.smithi077@0(leader) e2 *** Got Signal Terminated *** 2024-01-13T09:04:23.301 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Jan 13 09:04:22 smithi077 bash[70835]: ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi077 2024-01-13T09:04:23.313 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-99e41a38-b1f1-11ee-95ac-87774f69a715@mon.smithi077.service' 2024-01-13T09:04:23.358 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-13T09:04:23.358 INFO:tasks.cephadm.mon.smithi077:Stopped mon.smithi077 2024-01-13T09:04:23.358 INFO:tasks.cephadm.mon.smithi141:Stopping mon.smithi141... 2024-01-13T09:04:23.359 DEBUG:teuthology.orchestra.run.smithi141:> sudo systemctl stop ceph-99e41a38-b1f1-11ee-95ac-87774f69a715@mon.smithi141 2024-01-13T09:04:23.878 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:04:23 smithi141 systemd[1]: Stopping Ceph mon.smithi141 for 99e41a38-b1f1-11ee-95ac-87774f69a715... 2024-01-13T09:04:23.879 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:04:23 smithi141 bash[38854]: Error: no container with name or ID "ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon.smithi141" found: no such container 2024-01-13T09:04:23.879 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:04:23 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T09:04:23.625+0000 7f6333bc2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi141 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-13T09:04:23.879 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:04:23 smithi141 ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141[34416]: debug 2024-01-13T09:04:23.625+0000 7f6333bc2700 -1 mon.smithi141@1(peon) e2 *** Got Signal Terminated *** 2024-01-13T09:04:24.378 INFO:journalctl@ceph.mon.smithi141.smithi141.stdout:Jan 13 09:04:24 smithi141 bash[38893]: ceph-99e41a38-b1f1-11ee-95ac-87774f69a715-mon-smithi141 2024-01-13T09:04:24.400 DEBUG:teuthology.orchestra.run.smithi141:> sudo pkill -f 'journalctl -f -n 0 -u ceph-99e41a38-b1f1-11ee-95ac-87774f69a715@mon.smithi141.service' 2024-01-13T09:04:24.454 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-13T09:04:24.455 INFO:tasks.cephadm.mon.smithi141:Stopped mon.smithi141 2024-01-13T09:04:24.455 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 --force --keep-logs 2024-01-13T09:04:29.976 DEBUG:teuthology.orchestra.run.smithi141:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 --force --keep-logs 2024-01-13T09:04:32.913 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-13T09:04:32.952 DEBUG:teuthology.orchestra.run.smithi141:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-13T09:04:32.983 INFO:tasks.cephadm:Archiving crash dumps... 2024-01-13T09:04:32.986 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515791/remote/ubuntu@smithi077.front.sepia.ceph.com/crash 2024-01-13T09:04:32.988 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash -- . 2024-01-13T09:04:33.052 INFO:teuthology.orchestra.run.smithi077.stderr:tar: /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash: Cannot open: No such file or directory 2024-01-13T09:04:33.052 INFO:teuthology.orchestra.run.smithi077.stderr:tar: Error is not recoverable: exiting now 2024-01-13T09:04:33.054 DEBUG:teuthology.misc:Transferring archived files from smithi141:/var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515791/remote/ubuntu@smithi141.front.sepia.ceph.com/crash 2024-01-13T09:04:33.055 DEBUG:teuthology.orchestra.run.smithi141:> sudo tar c -f - -C /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash -- . 2024-01-13T09:04:33.087 INFO:teuthology.orchestra.run.smithi141.stderr:tar: /var/lib/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/crash: Cannot open: No such file or directory 2024-01-13T09:04:33.087 INFO:teuthology.orchestra.run.smithi141.stderr:tar: Error is not recoverable: exiting now 2024-01-13T09:04:33.088 INFO:tasks.cephadm:Checking cluster log for badness... 2024-01-13T09:04:33.088 DEBUG:teuthology.orchestra.run.smithi077:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-01-13T09:04:33.117 INFO:teuthology.orchestra.run.smithi077.stderr:grep: /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/ceph.log: No such file or directory 2024-01-13T09:04:33.118 INFO:tasks.cephadm:Compressing logs... 2024-01-13T09:04:33.118 DEBUG:teuthology.orchestra.run.smithi077:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-13T09:04:33.161 DEBUG:teuthology.orchestra.run.smithi141:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-13T09:04:33.190 INFO:teuthology.orchestra.run.smithi077.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-01-13T09:04:33.190 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-01-13T09:04:33.190 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/ceph-mon.smithi077.log 2024-01-13T09:04:33.190 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/ceph-volume.log 2024-01-13T09:04:33.191 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/ceph-mon.smithi077.log: gzip -5 --verbose -- /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/ceph-mgr.smithi077.amvfyi.log 2024-01-13T09:04:33.193 INFO:teuthology.orchestra.run.smithi141.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-01-13T09:04:33.193 INFO:teuthology.orchestra.run.smithi141.stderr:: No such file or directory 2024-01-13T09:04:33.193 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/ceph-mon.smithi141.log 2024-01-13T09:04:33.193 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/ceph-volume.log 2024-01-13T09:04:33.193 INFO:teuthology.orchestra.run.smithi141.stderr:gzip -5 --verbose -- /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/ceph-mgr.smithi141.ujrxla.log 2024-01-13T09:04:33.193 INFO:teuthology.orchestra.run.smithi141.stderr:/var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/ceph-mon.smithi141.log: /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/ceph-volume.log: /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/ceph-mgr.smithi141.ujrxla.log: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-01-13T09:04:33.194 INFO:teuthology.orchestra.run.smithi141.stderr: 91.3% -- replaced with /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/ceph-mgr.smithi141.ujrxla.log.gz 2024-01-13T09:04:33.195 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/ceph-volume.log: /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/ceph-mgr.smithi077.amvfyi.log: 93.9% -- replaced with /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/ceph-volume.log.gz 2024-01-13T09:04:33.195 INFO:teuthology.orchestra.run.smithi141.stderr: 94.0% -- replaced with /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/ceph-volume.log.gz 2024-01-13T09:04:33.198 INFO:teuthology.orchestra.run.smithi077.stderr: 92.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-01-13T09:04:33.252 INFO:teuthology.orchestra.run.smithi141.stderr: 92.5% -- replaced with /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/ceph-mon.smithi141.log.gz 2024-01-13T09:04:33.253 INFO:teuthology.orchestra.run.smithi141.stderr: 2024-01-13T09:04:33.254 INFO:teuthology.orchestra.run.smithi141.stderr:real 0m0.076s 2024-01-13T09:04:33.254 INFO:teuthology.orchestra.run.smithi141.stderr:user 0m0.084s 2024-01-13T09:04:33.254 INFO:teuthology.orchestra.run.smithi141.stderr:sys 0m0.012s 2024-01-13T09:04:33.275 INFO:teuthology.orchestra.run.smithi077.stderr: 90.7% -- replaced with /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/ceph-mgr.smithi077.amvfyi.log.gz 2024-01-13T09:04:33.405 INFO:teuthology.orchestra.run.smithi077.stderr: 89.9% -- replaced with /var/log/ceph/99e41a38-b1f1-11ee-95ac-87774f69a715/ceph-mon.smithi077.log.gz 2024-01-13T09:04:33.407 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-01-13T09:04:33.407 INFO:teuthology.orchestra.run.smithi077.stderr:real 0m0.230s 2024-01-13T09:04:33.407 INFO:teuthology.orchestra.run.smithi077.stderr:user 0m0.313s 2024-01-13T09:04:33.407 INFO:teuthology.orchestra.run.smithi077.stderr:sys 0m0.025s 2024-01-13T09:04:33.407 INFO:tasks.cephadm:Archiving logs... 2024-01-13T09:04:33.407 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515791/remote/ubuntu@smithi077.front.sepia.ceph.com/log 2024-01-13T09:04:33.408 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-13T09:04:33.536 DEBUG:teuthology.misc:Transferring archived files from smithi141:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515791/remote/ubuntu@smithi141.front.sepia.ceph.com/log 2024-01-13T09:04:33.537 DEBUG:teuthology.orchestra.run.smithi141:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-13T09:04:33.578 INFO:tasks.cephadm:Removing cluster... 2024-01-13T09:04:33.578 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 --force 2024-01-13T09:04:33.939 DEBUG:teuthology.orchestra.run.smithi141:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 99e41a38-b1f1-11ee-95ac-87774f69a715 --force 2024-01-13T09:04:34.304 INFO:tasks.cephadm:Removing cephadm ... 2024-01-13T09:04:34.305 DEBUG:teuthology.orchestra.run.smithi077:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-13T09:04:34.322 DEBUG:teuthology.orchestra.run.smithi141:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-13T09:04:34.339 INFO:tasks.cephadm:Teardown complete 2024-01-13T09:04:34.340 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 779, in ceph_osds while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (121) after waiting for 120 seconds 2024-01-13T09:04:34.561 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=492d3ce418f94becbb3de9d54ba6eee0 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 779, in ceph_osds while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (121) after waiting for 120 seconds 2024-01-13T09:04:34.564 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-01-13T09:04:34.574 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop 2024-01-13T09:04:34.581 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi077:/dev/vg_nvme/lv_1... 2024-01-13T09:04:34.582 DEBUG:teuthology.orchestra.run.smithi077:> sudo nvme disconnect -n lv_1 2024-01-13T09:04:34.704 INFO:teuthology.orchestra.run.smithi077.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-01-13T09:04:34.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T09:04:34.705 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi077:/dev/vg_nvme/lv_2... 2024-01-13T09:04:34.705 DEBUG:teuthology.orchestra.run.smithi077:> sudo nvme disconnect -n lv_2 2024-01-13T09:04:34.831 INFO:teuthology.orchestra.run.smithi077.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-01-13T09:04:34.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T09:04:34.832 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi077:/dev/vg_nvme/lv_3... 2024-01-13T09:04:34.832 DEBUG:teuthology.orchestra.run.smithi077:> sudo nvme disconnect -n lv_3 2024-01-13T09:04:34.956 INFO:teuthology.orchestra.run.smithi077.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-01-13T09:04:34.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T09:04:34.957 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi077:/dev/vg_nvme/lv_4... 2024-01-13T09:04:34.957 DEBUG:teuthology.orchestra.run.smithi077:> sudo nvme disconnect -n lv_4 2024-01-13T09:04:35.086 INFO:teuthology.orchestra.run.smithi077.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-01-13T09:04:35.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T09:04:35.087 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-01-13T09:04:35.087 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/scratch_devs 2024-01-13T09:04:35.115 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi141:/dev/vg_nvme/lv_1... 2024-01-13T09:04:35.115 DEBUG:teuthology.orchestra.run.smithi141:> sudo nvme disconnect -n lv_1 2024-01-13T09:04:35.256 INFO:teuthology.orchestra.run.smithi141.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-01-13T09:04:35.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T09:04:35.258 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi141:/dev/vg_nvme/lv_2... 2024-01-13T09:04:35.258 DEBUG:teuthology.orchestra.run.smithi141:> sudo nvme disconnect -n lv_2 2024-01-13T09:04:35.387 INFO:teuthology.orchestra.run.smithi141.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-01-13T09:04:35.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T09:04:35.390 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi141:/dev/vg_nvme/lv_3... 2024-01-13T09:04:35.390 DEBUG:teuthology.orchestra.run.smithi141:> sudo nvme disconnect -n lv_3 2024-01-13T09:04:35.513 INFO:teuthology.orchestra.run.smithi141.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-01-13T09:04:35.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T09:04:35.515 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi141:/dev/vg_nvme/lv_4... 2024-01-13T09:04:35.515 DEBUG:teuthology.orchestra.run.smithi141:> sudo nvme disconnect -n lv_4 2024-01-13T09:04:35.643 INFO:teuthology.orchestra.run.smithi141.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-01-13T09:04:35.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T09:04:35.645 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2024-01-13T09:04:35.645 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd of=/scratch_devs 2024-01-13T09:04:35.679 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-13T09:04:35.688 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-13T09:04:35.688 DEBUG:teuthology.orchestra.run.smithi077:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-13T09:04:35.691 DEBUG:teuthology.orchestra.run.smithi141:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-13T09:04:35.708 INFO:teuthology.orchestra.run.smithi077.stderr:bash: ntpq: command not found 2024-01-13T09:04:35.714 INFO:teuthology.orchestra.run.smithi077.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-13T09:04:35.715 INFO:teuthology.orchestra.run.smithi077.stdout:=============================================================================== 2024-01-13T09:04:35.715 INFO:teuthology.orchestra.run.smithi077.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 30 -664us[ -664us] +/- 72ms 2024-01-13T09:04:35.715 INFO:teuthology.orchestra.run.smithi077.stdout:^* hv02.front.sepia.ceph.com 3 6 377 30 +398us[ +406us] +/- 60ms 2024-01-13T09:04:35.715 INFO:teuthology.orchestra.run.smithi077.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 28 +193us[ +193us] +/- 78ms 2024-01-13T09:04:35.715 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T09:04:35.735 INFO:teuthology.orchestra.run.smithi141.stderr:bash: ntpq: command not found 2024-01-13T09:04:35.739 INFO:teuthology.orchestra.run.smithi141.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-13T09:04:35.739 INFO:teuthology.orchestra.run.smithi141.stdout:=============================================================================== 2024-01-13T09:04:35.739 INFO:teuthology.orchestra.run.smithi141.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 30 -642us[ -642us] +/- 72ms 2024-01-13T09:04:35.739 INFO:teuthology.orchestra.run.smithi141.stdout:^* hv02.front.sepia.ceph.com 3 6 377 33 +410us[ +450us] +/- 60ms 2024-01-13T09:04:35.739 INFO:teuthology.orchestra.run.smithi141.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 30 +214us[ +214us] +/- 77ms 2024-01-13T09:04:35.740 INFO:teuthology.orchestra.run.smithi141.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T09:04:35.741 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-13T09:04:35.755 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-13T09:04:35.756 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-13T09:04:35.767 DEBUG:teuthology.orchestra.run.smithi077:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-13T09:04:35.837 DEBUG:teuthology.orchestra.run.smithi141:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-13T09:04:35.897 DEBUG:teuthology.orchestra.run.smithi077:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-13T09:04:35.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T09:04:35.929 DEBUG:teuthology.orchestra.run.smithi141:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-13T09:04:35.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T09:04:35.958 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-13T09:04:35.967 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-13T09:04:35.978 INFO:teuthology.task.internal:Duration was 1027.824546 seconds 2024-01-13T09:04:35.978 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-13T09:04:35.990 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-13T09:04:35.990 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-13T09:04:35.993 DEBUG:teuthology.orchestra.run.smithi141:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-13T09:04:36.038 INFO:teuthology.orchestra.run.smithi077.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-13T09:04:36.044 INFO:teuthology.orchestra.run.smithi141.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-13T09:04:36.391 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-13T09:04:36.391 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi077.front.sepia.ceph.com 2024-01-13T09:04:36.391 DEBUG:teuthology.orchestra.run.smithi077:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-13T09:04:36.418 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi141.front.sepia.ceph.com 2024-01-13T09:04:36.419 DEBUG:teuthology.orchestra.run.smithi141:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-13T09:04:36.441 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-13T09:04:36.441 DEBUG:teuthology.orchestra.run.smithi077:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-13T09:04:36.461 DEBUG:teuthology.orchestra.run.smithi141:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-13T09:04:36.534 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-13T09:04:36.534 DEBUG:teuthology.orchestra.run.smithi077:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-13T09:04:36.537 DEBUG:teuthology.orchestra.run.smithi141:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-13T09:04:36.709 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-13T09:04:36.719 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-13T09:04:36.720 DEBUG:teuthology.orchestra.run.smithi077:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-13T09:04:36.747 DEBUG:teuthology.orchestra.run.smithi141:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-13T09:04:36.774 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-13T09:04:36.784 DEBUG:teuthology.orchestra.run.smithi077:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-13T09:04:36.789 DEBUG:teuthology.orchestra.run.smithi141:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-13T09:04:36.813 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = core 2024-01-13T09:04:36.840 INFO:teuthology.orchestra.run.smithi141.stdout:kernel.core_pattern = core 2024-01-13T09:04:36.861 DEBUG:teuthology.orchestra.run.smithi077:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-13T09:04:36.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T09:04:36.892 DEBUG:teuthology.orchestra.run.smithi141:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-13T09:04:36.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T09:04:36.918 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-13T09:04:36.928 INFO:teuthology.task.internal:Transferring archived files... 2024-01-13T09:04:36.928 DEBUG:teuthology.misc:Transferring archived files from smithi077:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515791/remote/smithi077 2024-01-13T09:04:36.929 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-13T09:04:36.976 DEBUG:teuthology.misc:Transferring archived files from smithi141:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515791/remote/smithi141 2024-01-13T09:04:36.977 DEBUG:teuthology.orchestra.run.smithi141:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-13T09:04:37.020 INFO:teuthology.task.internal:Removing archive directory... 2024-01-13T09:04:37.020 DEBUG:teuthology.orchestra.run.smithi077:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-13T09:04:37.022 DEBUG:teuthology.orchestra.run.smithi141:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-13T09:04:37.071 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-13T09:04:37.083 INFO:teuthology.task.internal:Not uploading archives. 2024-01-13T09:04:37.083 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-13T09:04:37.093 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-13T09:04:37.093 DEBUG:teuthology.orchestra.run.smithi077:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-13T09:04:37.095 DEBUG:teuthology.orchestra.run.smithi141:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-13T09:04:37.110 INFO:teuthology.orchestra.run.smithi077.stdout: 262175 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 13 09:04 /home/ubuntu/cephtest 2024-01-13T09:04:37.128 INFO:teuthology.orchestra.run.smithi141.stdout: 262175 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 13 09:04 /home/ubuntu/cephtest 2024-01-13T09:04:37.129 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-13T09:04:37.140 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-13T09:04:37.169 INFO:teuthology.nuke:Checking targets against current locks 2024-01-13T09:04:37.195 DEBUG:teuthology.nuke:shortname: smithi077 2024-01-13T09:04:37.196 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-13T09:04:37.220 DEBUG:teuthology.nuke:shortname: smithi141 2024-01-13T09:04:37.220 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-13T09:04:37.224 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515791', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-13 08:42:03.140021', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN8NLaKSd0BQrCfI69gFZWN0W/k4ulGQUWWPxeP9PPBdluMvnUoDRb0h4DX/BuL19Qag4kCSgSg5VrT/vRA2deg='} 2024-01-13T09:04:37.248 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi141.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515791', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-13 08:42:03.142064', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDAeiwsfWiWyk0KzxhIy3Ic4FDNhZ1DIHGf/d6PpuegFVnBW9OVMS6QlEN4luyqgl/19FSjxP7hQsEQa4BneOyU='} 2024-01-13T09:04:37.280 INFO:teuthology.orchestra.console.smithi077:Power off 2024-01-13T09:04:37.280 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-13T09:04:37.304 DEBUG:teuthology.orchestra.console.smithi077:power off output: Chassis Power Control: Down/Off 2024-01-13T09:04:37.306 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-13T09:04:37.322 INFO:teuthology.orchestra.console.smithi141:Power off 2024-01-13T09:04:37.322 DEBUG:teuthology.orchestra.console.smithi141:pexpect command: ipmitool -H smithi141.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-13T09:04:37.332 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on 2024-01-13T09:04:37.345 DEBUG:teuthology.orchestra.console.smithi141:power off output: Chassis Power Control: Down/Off 2024-01-13T09:04:37.345 DEBUG:teuthology.orchestra.console.smithi141:pexpect command: ipmitool -H smithi141.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-13T09:04:37.369 DEBUG:teuthology.orchestra.console.smithi141:check power output: Chassis Power is on 2024-01-13T09:04:41.333 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-13T09:04:41.370 DEBUG:teuthology.orchestra.console.smithi141:pexpect command: ipmitool -H smithi141.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-13T09:04:41.445 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on 2024-01-13T09:04:41.480 DEBUG:teuthology.orchestra.console.smithi141:check power output: Chassis Power is on 2024-01-13T09:04:45.449 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-13T09:04:45.481 DEBUG:teuthology.orchestra.console.smithi141:pexpect command: ipmitool -H smithi141.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-13T09:04:45.560 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on 2024-01-13T09:04:45.590 DEBUG:teuthology.orchestra.console.smithi141:check power output: Chassis Power is off 2024-01-13T09:04:45.690 INFO:teuthology.orchestra.console.smithi141:Power off completed 2024-01-13T09:04:49.564 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-13T09:04:49.674 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is off 2024-01-13T09:04:49.775 INFO:teuthology.orchestra.console.smithi077:Power off completed 2024-01-13T09:04:49.879 INFO:teuthology.run:Summary data: description: rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/repave-all} duration: 1027.824546098709 failure_reason: reached maximum tries (121) after waiting for 120 seconds owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=492d3ce418f94becbb3de9d54ba6eee0 status: fail success: false 2024-01-13T09:04:49.879 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-13T09:04:49.973 INFO:teuthology.run:FAIL