2024-01-13T08:05:55.235 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-13T08:05:55.236 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515725 --verbose --description rados/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515725/orig.config.yaml 2024-01-13T08:05:55.273 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-13T08:05:55.357 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515725 branch: wip-yuri2-testing-2024-01-12-1128-pacific description: rados/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} email: yweinste@redhat.com first_in_suite: false job_id: '7515725' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-yuri2-testing-2024-01-12-1128-pacific ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 445562ab4bc3ddfb386936119050695810860bcb ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 445562ab4bc3ddfb386936119050695810860bcb selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri2-testing-2024-01-12-1128-pacific sha1: 445562ab4bc3ddfb386936119050695810860bcb owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - - host.b - client.1 seed: 5246 sha1: 445562ab4bc3ddfb386936119050695810860bcb sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri2-testing-2024-01-12-1128-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 445562ab4bc3ddfb386936119050695810860bcb targets: smithi099.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGf7JtgD1eHBuWcCD/jYn9Ug/25uiQOIu51R/HNGaetszbYiIviq+iiBCD8rF9ANJ0wOhRLXdavAViwRt2KnYsw= smithi131.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFStCvrqjeTn1EnjfVBkPOdg++w7f7vX/2Ko2F4wPq9kDMHOP3I1aBrUEPOcyvXM2RoVC9l2fc2NItarP+zIN/o= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:3.0 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - nvme_loop: null - cephadm: roleless: true - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - vip: null - cephadm.shell: host.a: - ceph orch device ls --refresh - vip.exec: all-hosts: - systemctl stop nfs-server - cephadm.shell: host.a: - ceph fs volume create foofs - ceph nfs cluster create foo --ingress --virtual-ip {{VIP0}}/{{VIPPREFIXLEN}} --port 2999 - ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake - cephadm.wait_for_service: service: nfs.foo - cephadm.wait_for_service: service: ingress.nfs.foo - vip.exec: host.a: - mkdir /mnt/foo - sleep 5 - mount -t nfs {{VIP0}}:/fake /mnt/foo -o port=2999 - echo test > /mnt/foo/testfile - sync - cephadm.shell: host.a: - "echo \"Check with each haproxy down in turn...\"\nfor haproxy in `ceph orch\ \ ps | grep ^haproxy.nfs.foo. | awk '{print $1}'`; do\n ceph orch daemon stop\ \ $haproxy\n while ! ceph orch ps | grep $haproxy | grep stopped; do sleep\ \ 1 ; done\n cat /mnt/foo/testfile\n echo $haproxy > /mnt/foo/testfile\n \ \ sync\n ceph orch daemon start $haproxy\n while ! ceph orch ps | grep $haproxy\ \ | grep running; do sleep 1 ; done\ndone\n" volumes: - /mnt/foo:/mnt/foo - vip.exec: all-hosts: - "echo \"Check with $(hostname) ganesha(s) down...\"\nfor c in `systemctl | grep\ \ ceph- | grep @nfs | awk '{print $1}'`; do\n cid=`echo $c | sed 's/@/-/'`\n\ \ id=`echo $c | cut -d @ -f 2 | sed 's/.service$//'`\n fsid=`echo $c |\ \ cut -d @ -f 1 | cut -d - -f 2-`\n echo \"Removing daemon $id fsid $fsid...\"\ \n sudo $TESTDIR/cephadm rm-daemon --fsid $fsid --name $id\n\n echo \"\ Waking up cephadm...\"\n sudo $TESTDIR/cephadm shell -- ceph orch ps --refresh\n\ \n while ! timeout 1 cat /mnt/foo/testfile ; do true ; done\n echo \"\ Mount is back!\"\ndone\n" - cephadm.shell: host.a: - stat -c '%u %g' /var/log/ceph | grep '167 167' - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch ls | grep '^osd.all-available-devices ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-12_21:52:25 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-01-13T08:05:55.358 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa; will attempt to use it 2024-01-13T08:05:55.358 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks 2024-01-13T08:05:55.359 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-13T08:05:55.360 INFO:teuthology.task.internal:Checking packages... 2024-01-13T08:05:55.385 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '445562ab4bc3ddfb386936119050695810860bcb' 2024-01-13T08:05:55.385 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-13T08:05:55.385 INFO:teuthology.packaging:ref: None 2024-01-13T08:05:55.385 INFO:teuthology.packaging:tag: None 2024-01-13T08:05:55.385 INFO:teuthology.packaging:branch: wip-yuri2-testing-2024-01-12-1128-pacific 2024-01-13T08:05:55.385 INFO:teuthology.packaging:sha1: 445562ab4bc3ddfb386936119050695810860bcb 2024-01-13T08:05:55.385 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri2-testing-2024-01-12-1128-pacific 2024-01-13T08:05:55.533 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-456.g445562ab 2024-01-13T08:05:55.535 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-13T08:05:55.541 INFO:teuthology.task.internal:no buildpackages task found 2024-01-13T08:05:55.541 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-13T08:05:55.579 INFO:teuthology.task.internal:Saving configuration 2024-01-13T08:05:55.592 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-13T08:05:55.603 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-13T08:05:55.628 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi099.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515725', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-13 08:00:42.270691', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGf7JtgD1eHBuWcCD/jYn9Ug/25uiQOIu51R/HNGaetszbYiIviq+iiBCD8rF9ANJ0wOhRLXdavAViwRt2KnYsw='} 2024-01-13T08:05:55.650 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi131.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515725', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-13 08:00:42.273198', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFStCvrqjeTn1EnjfVBkPOdg++w7f7vX/2Ko2F4wPq9kDMHOP3I1aBrUEPOcyvXM2RoVC9l2fc2NItarP+zIN/o='} 2024-01-13T08:05:55.650 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-13T08:05:55.656 INFO:teuthology.task.internal:roles: ubuntu@smithi099.front.sepia.ceph.com - ['host.a', 'client.0'] 2024-01-13T08:05:55.656 INFO:teuthology.task.internal:roles: ubuntu@smithi131.front.sepia.ceph.com - ['host.b', 'client.1'] 2024-01-13T08:05:55.656 INFO:teuthology.run_tasks:Running task console_log... 2024-01-13T08:05:55.749 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd3d77c0160>, signals=[15]) 2024-01-13T08:05:55.749 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-13T08:05:55.755 INFO:teuthology.task.internal:Opening connections... 2024-01-13T08:05:55.755 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi099.front.sepia.ceph.com 2024-01-13T08:05:55.757 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-13T08:05:55.830 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi131.front.sepia.ceph.com 2024-01-13T08:05:55.831 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-13T08:05:55.904 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-13T08:05:55.910 DEBUG:teuthology.orchestra.run.smithi099:> uname -m 2024-01-13T08:05:55.929 INFO:teuthology.orchestra.run.smithi099.stdout:x86_64 2024-01-13T08:05:55.930 DEBUG:teuthology.orchestra.run.smithi099:> cat /etc/os-release 2024-01-13T08:05:55.988 INFO:teuthology.orchestra.run.smithi099.stdout:NAME="Red Hat Enterprise Linux" 2024-01-13T08:05:55.989 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION="8.6 (Ootpa)" 2024-01-13T08:05:55.989 INFO:teuthology.orchestra.run.smithi099.stdout:ID="rhel" 2024-01-13T08:05:55.989 INFO:teuthology.orchestra.run.smithi099.stdout:ID_LIKE="fedora" 2024-01-13T08:05:55.989 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION_ID="8.6" 2024-01-13T08:05:55.989 INFO:teuthology.orchestra.run.smithi099.stdout:PLATFORM_ID="platform:el8" 2024-01-13T08:05:55.989 INFO:teuthology.orchestra.run.smithi099.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-01-13T08:05:55.989 INFO:teuthology.orchestra.run.smithi099.stdout:ANSI_COLOR="0;31" 2024-01-13T08:05:55.989 INFO:teuthology.orchestra.run.smithi099.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-01-13T08:05:55.989 INFO:teuthology.orchestra.run.smithi099.stdout:HOME_URL="https://www.redhat.com/" 2024-01-13T08:05:55.989 INFO:teuthology.orchestra.run.smithi099.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-01-13T08:05:55.990 INFO:teuthology.orchestra.run.smithi099.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-13T08:05:55.990 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:05:55.990 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-13T08:05:55.990 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-01-13T08:05:55.990 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-01-13T08:05:55.990 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-01-13T08:05:55.991 INFO:teuthology.lock.ops:Updating smithi099.front.sepia.ceph.com on lock server 2024-01-13T08:05:56.017 DEBUG:teuthology.orchestra.run.smithi131:> uname -m 2024-01-13T08:05:56.038 INFO:teuthology.orchestra.run.smithi131.stdout:x86_64 2024-01-13T08:05:56.038 DEBUG:teuthology.orchestra.run.smithi131:> cat /etc/os-release 2024-01-13T08:05:56.097 INFO:teuthology.orchestra.run.smithi131.stdout:NAME="Red Hat Enterprise Linux" 2024-01-13T08:05:56.098 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION="8.6 (Ootpa)" 2024-01-13T08:05:56.098 INFO:teuthology.orchestra.run.smithi131.stdout:ID="rhel" 2024-01-13T08:05:56.098 INFO:teuthology.orchestra.run.smithi131.stdout:ID_LIKE="fedora" 2024-01-13T08:05:56.098 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION_ID="8.6" 2024-01-13T08:05:56.098 INFO:teuthology.orchestra.run.smithi131.stdout:PLATFORM_ID="platform:el8" 2024-01-13T08:05:56.098 INFO:teuthology.orchestra.run.smithi131.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-01-13T08:05:56.098 INFO:teuthology.orchestra.run.smithi131.stdout:ANSI_COLOR="0;31" 2024-01-13T08:05:56.098 INFO:teuthology.orchestra.run.smithi131.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-01-13T08:05:56.098 INFO:teuthology.orchestra.run.smithi131.stdout:HOME_URL="https://www.redhat.com/" 2024-01-13T08:05:56.098 INFO:teuthology.orchestra.run.smithi131.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-01-13T08:05:56.098 INFO:teuthology.orchestra.run.smithi131.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-13T08:05:56.099 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-01-13T08:05:56.099 INFO:teuthology.orchestra.run.smithi131.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-13T08:05:56.099 INFO:teuthology.orchestra.run.smithi131.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-01-13T08:05:56.099 INFO:teuthology.orchestra.run.smithi131.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-01-13T08:05:56.099 INFO:teuthology.orchestra.run.smithi131.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-01-13T08:05:56.099 INFO:teuthology.lock.ops:Updating smithi131.front.sepia.ceph.com on lock server 2024-01-13T08:05:56.117 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-13T08:05:56.125 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-13T08:05:56.153 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-13T08:05:56.153 DEBUG:teuthology.orchestra.run.smithi099:> test '!' -e /home/ubuntu/cephtest 2024-01-13T08:05:56.155 DEBUG:teuthology.orchestra.run.smithi131:> test '!' -e /home/ubuntu/cephtest 2024-01-13T08:05:56.174 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-13T08:05:56.179 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-13T08:05:56.180 DEBUG:teuthology.orchestra.run.smithi099:> test -z $(ls -A /var/lib/ceph) 2024-01-13T08:05:56.214 DEBUG:teuthology.orchestra.run.smithi131:> test -z $(ls -A /var/lib/ceph) 2024-01-13T08:05:56.248 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-13T08:05:56.306 INFO:teuthology.run_tasks:Running task kernel... 2024-01-13T08:05:56.320 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-01-13T08:05:56.320 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-01-13T08:05:56.321 DEBUG:teuthology.orchestra.run.smithi099:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-13T08:05:56.321 DEBUG:teuthology.orchestra.run.smithi131:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-13T08:05:56.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T08:05:56.340 DEBUG:teuthology.orchestra.run.smithi099:> uname -r 2024-01-13T08:05:56.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T08:05:56.341 DEBUG:teuthology.orchestra.run.smithi131:> uname -r 2024-01-13T08:05:56.398 INFO:teuthology.orchestra.run.smithi099.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-01-13T08:05:56.398 INFO:teuthology.task.kernel:Running kernel on smithi099: 4.18.0-513.9.1.el8_9.x86_64 2024-01-13T08:05:56.398 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum install -y kernel 2024-01-13T08:05:56.400 INFO:teuthology.orchestra.run.smithi131.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-01-13T08:05:56.401 INFO:teuthology.task.kernel:Running kernel on smithi131: 4.18.0-513.9.1.el8_9.x86_64 2024-01-13T08:05:56.401 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum install -y kernel 2024-01-13T08:05:57.854 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-01-13T08:05:57.854 INFO:teuthology.orchestra.run.smithi099.stdout:Unable to read consumer identity 2024-01-13T08:05:58.006 INFO:teuthology.orchestra.run.smithi131.stdout:Updating Subscription Management repositories. 2024-01-13T08:05:58.006 INFO:teuthology.orchestra.run.smithi131.stdout:Unable to read consumer identity 2024-01-13T08:05:58.162 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:05:58.163 INFO:teuthology.orchestra.run.smithi099.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-13T08:05:58.163 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:05:58.273 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-01-13T08:05:58.273 INFO:teuthology.orchestra.run.smithi131.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-13T08:05:58.273 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-01-13T08:05:59.973 INFO:teuthology.orchestra.run.smithi099.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2024-01-13T08:06:00.246 INFO:teuthology.orchestra.run.smithi131.stdout:Copr repo for python3-asyncssh owned by ceph 19 kB/s | 3.5 kB 00:00 2024-01-13T08:06:02.132 INFO:teuthology.orchestra.run.smithi099.stdout:Extra Packages for Enterprise Linux 7.5 MB/s | 16 MB 00:02 2024-01-13T08:06:05.631 INFO:teuthology.orchestra.run.smithi099.stdout:lab-extras 629 kB/s | 24 kB 00:00 2024-01-13T08:06:05.996 INFO:teuthology.orchestra.run.smithi131.stdout:Extra Packages for Enterprise Linux 2.8 MB/s | 16 MB 00:05 2024-01-13T08:06:06.343 INFO:teuthology.orchestra.run.smithi099.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-01-13T08:06:06.343 INFO:teuthology.orchestra.run.smithi099.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-01-13T08:06:06.425 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-01-13T08:06:06.426 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2024-01-13T08:06:06.426 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-01-13T08:06:06.569 DEBUG:teuthology.orchestra.run.smithi099:> echo no | sudo yum reinstall kernel || true 2024-01-13T08:06:06.906 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-01-13T08:06:06.906 INFO:teuthology.orchestra.run.smithi099.stdout:Unable to read consumer identity 2024-01-13T08:06:07.126 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:06:07.127 INFO:teuthology.orchestra.run.smithi099.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-13T08:06:07.127 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:06:07.433 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:00:02 ago on Sat 13 Jan 2024 08:06:05 AM UTC. 2024-01-13T08:06:07.510 INFO:teuthology.orchestra.run.smithi099.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-01-13T08:06:07.517 INFO:teuthology.orchestra.run.smithi099.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-01-13T08:06:07.556 INFO:teuthology.orchestra.run.smithi099.stderr:Error: No packages marked for reinstall. 2024-01-13T08:06:07.603 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum reinstall -y kernel || true 2024-01-13T08:06:07.937 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-01-13T08:06:07.937 INFO:teuthology.orchestra.run.smithi099.stdout:Unable to read consumer identity 2024-01-13T08:06:08.124 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:06:08.124 INFO:teuthology.orchestra.run.smithi099.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-13T08:06:08.124 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:06:08.431 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:00:03 ago on Sat 13 Jan 2024 08:06:05 AM UTC. 2024-01-13T08:06:08.497 INFO:teuthology.orchestra.run.smithi099.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-01-13T08:06:08.498 INFO:teuthology.orchestra.run.smithi099.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-01-13T08:06:08.527 INFO:teuthology.orchestra.run.smithi099.stderr:Error: No packages marked for reinstall. 2024-01-13T08:06:08.579 DEBUG:teuthology.orchestra.run.smithi099:> rpm -q kernel | sort -rV | head -n 1 2024-01-13T08:06:08.738 INFO:teuthology.orchestra.run.smithi099.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-01-13T08:06:08.738 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-01-13T08:06:08.738 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-01-13T08:06:08.738 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-01-13T08:06:08.739 DEBUG:teuthology.orchestra.run.smithi099:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-13T08:06:08.772 INFO:teuthology.orchestra.run.smithi099.stdout:ttyS1 2024-01-13T08:06:08.791 DEBUG:teuthology.parallel:result is None 2024-01-13T08:06:09.621 INFO:teuthology.orchestra.run.smithi131.stdout:lab-extras 555 kB/s | 24 kB 00:00 2024-01-13T08:06:10.315 INFO:teuthology.orchestra.run.smithi131.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-01-13T08:06:10.316 INFO:teuthology.orchestra.run.smithi131.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-01-13T08:06:10.393 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2024-01-13T08:06:10.394 INFO:teuthology.orchestra.run.smithi131.stdout:Nothing to do. 2024-01-13T08:06:10.394 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2024-01-13T08:06:10.539 DEBUG:teuthology.orchestra.run.smithi131:> echo no | sudo yum reinstall kernel || true 2024-01-13T08:06:10.884 INFO:teuthology.orchestra.run.smithi131.stdout:Updating Subscription Management repositories. 2024-01-13T08:06:10.884 INFO:teuthology.orchestra.run.smithi131.stdout:Unable to read consumer identity 2024-01-13T08:06:11.134 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-01-13T08:06:11.134 INFO:teuthology.orchestra.run.smithi131.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-13T08:06:11.134 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-01-13T08:06:11.441 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:00:02 ago on Sat 13 Jan 2024 08:06:09 AM UTC. 2024-01-13T08:06:11.519 INFO:teuthology.orchestra.run.smithi131.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-01-13T08:06:11.520 INFO:teuthology.orchestra.run.smithi131.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-01-13T08:06:11.563 INFO:teuthology.orchestra.run.smithi131.stderr:Error: No packages marked for reinstall. 2024-01-13T08:06:11.611 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum reinstall -y kernel || true 2024-01-13T08:06:11.948 INFO:teuthology.orchestra.run.smithi131.stdout:Updating Subscription Management repositories. 2024-01-13T08:06:11.948 INFO:teuthology.orchestra.run.smithi131.stdout:Unable to read consumer identity 2024-01-13T08:06:12.189 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-01-13T08:06:12.189 INFO:teuthology.orchestra.run.smithi131.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-13T08:06:12.189 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-01-13T08:06:12.490 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:00:03 ago on Sat 13 Jan 2024 08:06:09 AM UTC. 2024-01-13T08:06:12.558 INFO:teuthology.orchestra.run.smithi131.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-01-13T08:06:12.558 INFO:teuthology.orchestra.run.smithi131.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-01-13T08:06:12.595 INFO:teuthology.orchestra.run.smithi131.stderr:Error: No packages marked for reinstall. 2024-01-13T08:06:12.647 DEBUG:teuthology.orchestra.run.smithi131:> rpm -q kernel | sort -rV | head -n 1 2024-01-13T08:06:12.798 INFO:teuthology.orchestra.run.smithi131.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-01-13T08:06:12.799 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-01-13T08:06:12.799 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-01-13T08:06:12.799 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-01-13T08:06:12.799 DEBUG:teuthology.orchestra.run.smithi131:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-13T08:06:12.832 INFO:teuthology.orchestra.run.smithi131.stdout:ttyS1 2024-01-13T08:06:12.845 DEBUG:teuthology.parallel:result is None 2024-01-13T08:06:12.845 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-13T08:06:12.853 INFO:teuthology.task.internal:Creating test directory... 2024-01-13T08:06:12.853 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-13T08:06:12.856 DEBUG:teuthology.orchestra.run.smithi131:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-13T08:06:12.902 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-13T08:06:12.908 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-13T08:06:12.936 INFO:teuthology.task.internal:Creating archive directory... 2024-01-13T08:06:12.936 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-13T08:06:12.938 DEBUG:teuthology.orchestra.run.smithi131:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-13T08:06:12.965 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-13T08:06:12.972 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-13T08:06:12.972 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-13T08:06:13.005 DEBUG:teuthology.orchestra.run.smithi131:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-13T08:06:13.033 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-13T08:06:13.037 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-13T08:06:13.044 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-13T08:06:13.049 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-13T08:06:13.051 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-13T08:06:13.058 INFO:teuthology.task.internal:Configuring sudo... 2024-01-13T08:06:13.058 DEBUG:teuthology.orchestra.run.smithi099:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-13T08:06:13.095 DEBUG:teuthology.orchestra.run.smithi131:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-13T08:06:13.124 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-13T08:06:13.133 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-13T08:06:13.133 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-13T08:06:13.166 DEBUG:teuthology.orchestra.run.smithi131:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-13T08:06:13.183 DEBUG:teuthology.orchestra.run.smithi099:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-13T08:06:13.270 DEBUG:teuthology.orchestra.run.smithi099:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-13T08:06:13.327 DEBUG:teuthology.orchestra.run.smithi099:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-13T08:06:13.409 DEBUG:teuthology.orchestra.run.smithi099:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-13T08:06:13.454 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-01-13T08:06:13.455 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-13T08:06:13.522 DEBUG:teuthology.orchestra.run.smithi131:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-13T08:06:13.561 DEBUG:teuthology.orchestra.run.smithi131:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-13T08:06:13.621 DEBUG:teuthology.orchestra.run.smithi131:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-13T08:06:13.705 DEBUG:teuthology.orchestra.run.smithi131:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-13T08:06:13.747 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-01-13T08:06:13.747 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-13T08:06:13.813 DEBUG:teuthology.orchestra.run.smithi099:> sudo service rsyslog restart 2024-01-13T08:06:13.815 DEBUG:teuthology.orchestra.run.smithi131:> sudo service rsyslog restart 2024-01-13T08:06:13.859 INFO:teuthology.orchestra.run.smithi099.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-13T08:06:13.897 INFO:teuthology.orchestra.run.smithi131.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-13T08:06:14.246 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-13T08:06:14.254 INFO:teuthology.task.internal:Starting timer... 2024-01-13T08:06:14.254 INFO:teuthology.run_tasks:Running task pcp... 2024-01-13T08:06:14.297 INFO:teuthology.run_tasks:Running task selinux... 2024-01-13T08:06:14.321 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-01-13T08:06:14.321 DEBUG:teuthology.orchestra.run.smithi099:> sudo service auditd rotate 2024-01-13T08:06:14.387 INFO:teuthology.orchestra.run.smithi099.stdout:Rotating logs: 2024-01-13T08:06:14.389 DEBUG:teuthology.orchestra.run.smithi131:> sudo service auditd rotate 2024-01-13T08:06:14.465 INFO:teuthology.orchestra.run.smithi131.stdout:Rotating logs: 2024-01-13T08:06:14.467 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-13T08:06:14.467 DEBUG:teuthology.orchestra.run.smithi099:> /usr/sbin/getenforce 2024-01-13T08:06:14.493 INFO:teuthology.orchestra.run.smithi099.stdout:Permissive 2024-01-13T08:06:14.493 DEBUG:teuthology.orchestra.run.smithi131:> /usr/sbin/getenforce 2024-01-13T08:06:14.516 INFO:teuthology.orchestra.run.smithi131.stdout:Permissive 2024-01-13T08:06:14.516 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi099.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi131.front.sepia.ceph.com': 'permissive'} 2024-01-13T08:06:14.516 DEBUG:teuthology.orchestra.run.smithi099:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-13T08:06:14.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T08:06:14.561 DEBUG:teuthology.orchestra.run.smithi131:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-13T08:06:14.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T08:06:14.588 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-13T08:06:14.588 DEBUG:teuthology.orchestra.run.smithi099:> sudo /usr/sbin/setenforce permissive 2024-01-13T08:06:14.629 DEBUG:teuthology.orchestra.run.smithi131:> sudo /usr/sbin/setenforce permissive 2024-01-13T08:06:14.660 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-13T08:06:14.672 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-13T08:06:14.690 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-13T08:06:14.692 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi099.front.sepia.ceph.com,smithi131.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-13T08:14:10.980 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi099.front.sepia.ceph.com'), Remote(name='ubuntu@smithi131.front.sepia.ceph.com')] 2024-01-13T08:14:10.981 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-01-13T08:14:10.982 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-13T08:14:11.058 DEBUG:teuthology.orchestra.run.smithi099:> true 2024-01-13T08:14:11.125 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-01-13T08:14:11.126 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-01-13T08:14:11.126 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-13T08:14:11.199 DEBUG:teuthology.orchestra.run.smithi131:> true 2024-01-13T08:14:11.269 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-01-13T08:14:11.269 INFO:teuthology.run_tasks:Running task clock... 2024-01-13T08:14:11.282 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-13T08:14:11.282 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-13T08:14:11.283 DEBUG:teuthology.orchestra.run.smithi099:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-13T08:14:11.288 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-13T08:14:11.288 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-13T08:14:11.316 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-13T08:14:11.331 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-13T08:14:11.345 INFO:teuthology.orchestra.run.smithi131.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-13T08:14:11.360 INFO:teuthology.orchestra.run.smithi131.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-13T08:14:11.397 INFO:teuthology.orchestra.run.smithi099.stderr:sudo: ntpd: command not found 2024-01-13T08:14:11.408 INFO:teuthology.orchestra.run.smithi099.stdout:506 Cannot talk to daemon 2024-01-13T08:14:11.422 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-13T08:14:11.435 INFO:teuthology.orchestra.run.smithi131.stderr:sudo: ntpd: command not found 2024-01-13T08:14:11.436 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-13T08:14:11.447 INFO:teuthology.orchestra.run.smithi131.stdout:506 Cannot talk to daemon 2024-01-13T08:14:11.462 INFO:teuthology.orchestra.run.smithi131.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-13T08:14:11.476 INFO:teuthology.orchestra.run.smithi131.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-13T08:14:11.518 INFO:teuthology.orchestra.run.smithi099.stderr:bash: ntpq: command not found 2024-01-13T08:14:11.522 INFO:teuthology.orchestra.run.smithi099.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-13T08:14:11.522 INFO:teuthology.orchestra.run.smithi099.stdout:=============================================================================== 2024-01-13T08:14:11.522 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T08:14:11.522 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T08:14:11.523 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T08:14:11.523 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T08:14:11.545 INFO:teuthology.orchestra.run.smithi131.stderr:bash: ntpq: command not found 2024-01-13T08:14:11.549 INFO:teuthology.orchestra.run.smithi131.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-13T08:14:11.549 INFO:teuthology.orchestra.run.smithi131.stdout:=============================================================================== 2024-01-13T08:14:11.550 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T08:14:11.550 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T08:14:11.550 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T08:14:11.550 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T08:14:11.552 INFO:teuthology.run_tasks:Running task pexec... 2024-01-13T08:14:11.563 INFO:teuthology.task.pexec:Executing custom commands... 2024-01-13T08:14:11.563 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi099.front.sepia.ceph.com 2024-01-13T08:14:11.563 DEBUG:teuthology.orchestra.run.smithi099:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-13T08:14:11.564 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi131.front.sepia.ceph.com 2024-01-13T08:14:11.564 DEBUG:teuthology.orchestra.run.smithi131:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-13T08:14:11.940 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-01-13T08:14:11.972 INFO:teuthology.orchestra.run.smithi131.stdout:Updating Subscription Management repositories. 2024-01-13T08:14:13.504 INFO:teuthology.orchestra.run.smithi131.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 39 kB/s | 4.5 kB 00:00 2024-01-13T08:14:13.610 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 34 kB/s | 4.5 kB 00:00 2024-01-13T08:14:14.090 INFO:teuthology.orchestra.run.smithi131.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 34 kB/s | 4.1 kB 00:00 2024-01-13T08:14:14.262 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 33 kB/s | 4.1 kB 00:00 2024-01-13T08:14:16.826 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2024-01-13T08:14:16.827 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-01-13T08:14:16.827 INFO:teuthology.orchestra.run.smithi131.stdout: Package Architecture Version Repository Size 2024-01-13T08:14:16.827 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-01-13T08:14:16.827 INFO:teuthology.orchestra.run.smithi131.stdout:Resetting modules: 2024-01-13T08:14:16.828 INFO:teuthology.orchestra.run.smithi131.stdout: container-tools 2024-01-13T08:14:16.828 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-01-13T08:14:16.828 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2024-01-13T08:14:16.828 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-01-13T08:14:16.828 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-01-13T08:14:17.009 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-01-13T08:14:17.010 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-01-13T08:14:17.010 INFO:teuthology.orchestra.run.smithi099.stdout: Package Architecture Version Repository Size 2024-01-13T08:14:17.010 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-01-13T08:14:17.010 INFO:teuthology.orchestra.run.smithi099.stdout:Resetting modules: 2024-01-13T08:14:17.010 INFO:teuthology.orchestra.run.smithi099.stdout: container-tools 2024-01-13T08:14:17.010 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:14:17.010 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-01-13T08:14:17.010 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-01-13T08:14:17.010 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:14:29.929 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2024-01-13T08:14:30.323 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-01-13T08:14:30.397 INFO:teuthology.orchestra.run.smithi131.stdout:Updating Subscription Management repositories. 2024-01-13T08:14:30.781 INFO:teuthology.orchestra.run.smithi099.stdout:Updating Subscription Management repositories. 2024-01-13T08:14:32.620 INFO:teuthology.orchestra.run.smithi131.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 37 kB/s | 4.5 kB 00:00 2024-01-13T08:14:32.662 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 41 kB/s | 4.5 kB 00:00 2024-01-13T08:14:33.233 INFO:teuthology.orchestra.run.smithi131.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 32 kB/s | 4.1 kB 00:00 2024-01-13T08:14:33.305 INFO:teuthology.orchestra.run.smithi099.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 37 kB/s | 4.1 kB 00:00 2024-01-13T08:14:35.911 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2024-01-13T08:14:35.942 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-01-13T08:14:36.059 INFO:teuthology.orchestra.run.smithi131.stdout:========================================================================================================================== 2024-01-13T08:14:36.059 INFO:teuthology.orchestra.run.smithi131.stdout: Package Arch Version Repository Size 2024-01-13T08:14:36.059 INFO:teuthology.orchestra.run.smithi131.stdout:========================================================================================================================== 2024-01-13T08:14:36.059 INFO:teuthology.orchestra.run.smithi131.stdout:Installing group/module packages: 2024-01-13T08:14:36.059 INFO:teuthology.orchestra.run.smithi131.stdout: buildah x86_64 1.19.9-6.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 6.6 M 2024-01-13T08:14:36.059 INFO:teuthology.orchestra.run.smithi131.stdout: cockpit-podman noarch 29-2.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 1.1 M 2024-01-13T08:14:36.059 INFO:teuthology.orchestra.run.smithi131.stdout: crun x86_64 0.18-3.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 185 k 2024-01-13T08:14:36.060 INFO:teuthology.orchestra.run.smithi131.stdout: skopeo x86_64 1:1.2.4-2.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 6.2 M 2024-01-13T08:14:36.060 INFO:teuthology.orchestra.run.smithi131.stdout: toolbox x86_64 0.0.99.3-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 2.2 M 2024-01-13T08:14:36.060 INFO:teuthology.orchestra.run.smithi131.stdout: udica noarch 0.2.4-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 51 k 2024-01-13T08:14:36.060 INFO:teuthology.orchestra.run.smithi131.stdout:Installing dependencies: 2024-01-13T08:14:36.060 INFO:teuthology.orchestra.run.smithi131.stdout: yajl x86_64 2.1.0-12.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2024-01-13T08:14:36.060 INFO:teuthology.orchestra.run.smithi131.stdout:Downgrading: 2024-01-13T08:14:36.060 INFO:teuthology.orchestra.run.smithi131.stdout: conmon x86_64 2:2.0.26-3.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 51 k 2024-01-13T08:14:36.060 INFO:teuthology.orchestra.run.smithi131.stdout: container-selinux noarch 2:2.189.0-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 55 k 2024-01-13T08:14:36.060 INFO:teuthology.orchestra.run.smithi131.stdout: containernetworking-plugins x86_64 0.9.1-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 19 M 2024-01-13T08:14:36.060 INFO:teuthology.orchestra.run.smithi131.stdout: containers-common x86_64 1:1.2.4-2.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 91 k 2024-01-13T08:14:36.060 INFO:teuthology.orchestra.run.smithi131.stdout: criu x86_64 3.15-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 511 k 2024-01-13T08:14:36.060 INFO:teuthology.orchestra.run.smithi131.stdout: fuse-overlayfs x86_64 1.4.0-2.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 72 k 2024-01-13T08:14:36.060 INFO:teuthology.orchestra.run.smithi131.stdout: libslirp x86_64 4.3.1-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 69 k 2024-01-13T08:14:36.061 INFO:teuthology.orchestra.run.smithi131.stdout: podman x86_64 3.0.1-15.module+el8.7.0+18021+1e286cc8 rhel-8-for-x86_64-appstream-rpms 11 M 2024-01-13T08:14:36.061 INFO:teuthology.orchestra.run.smithi131.stdout: podman-catatonit x86_64 3.0.1-15.module+el8.7.0+18021+1e286cc8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-01-13T08:14:36.061 INFO:teuthology.orchestra.run.smithi131.stdout: podman-docker noarch 3.0.1-15.module+el8.7.0+18021+1e286cc8 rhel-8-for-x86_64-appstream-rpms 57 k 2024-01-13T08:14:36.061 INFO:teuthology.orchestra.run.smithi131.stdout: runc x86_64 1.0.0-73.rc95.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 2.9 M 2024-01-13T08:14:36.061 INFO:teuthology.orchestra.run.smithi131.stdout: slirp4netns x86_64 1.1.8-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 51 k 2024-01-13T08:14:36.061 INFO:teuthology.orchestra.run.smithi131.stdout:Installing module profiles: 2024-01-13T08:14:36.061 INFO:teuthology.orchestra.run.smithi131.stdout: container-tools/common 2024-01-13T08:14:36.061 INFO:teuthology.orchestra.run.smithi131.stdout:Enabling module streams: 2024-01-13T08:14:36.061 INFO:teuthology.orchestra.run.smithi131.stdout: container-tools 3.0 2024-01-13T08:14:36.061 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-01-13T08:14:36.061 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2024-01-13T08:14:36.061 INFO:teuthology.orchestra.run.smithi131.stdout:========================================================================================================================== 2024-01-13T08:14:36.061 INFO:teuthology.orchestra.run.smithi131.stdout:Install 7 Packages 2024-01-13T08:14:36.061 INFO:teuthology.orchestra.run.smithi131.stdout:Downgrade 12 Packages 2024-01-13T08:14:36.062 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-01-13T08:14:36.062 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 50 M 2024-01-13T08:14:36.062 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2024-01-13T08:14:36.093 INFO:teuthology.orchestra.run.smithi099.stdout:========================================================================================================================== 2024-01-13T08:14:36.093 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-01-13T08:14:36.093 INFO:teuthology.orchestra.run.smithi099.stdout:========================================================================================================================== 2024-01-13T08:14:36.093 INFO:teuthology.orchestra.run.smithi099.stdout:Installing group/module packages: 2024-01-13T08:14:36.094 INFO:teuthology.orchestra.run.smithi099.stdout: buildah x86_64 1.19.9-6.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 6.6 M 2024-01-13T08:14:36.094 INFO:teuthology.orchestra.run.smithi099.stdout: cockpit-podman noarch 29-2.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 1.1 M 2024-01-13T08:14:36.094 INFO:teuthology.orchestra.run.smithi099.stdout: crun x86_64 0.18-3.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 185 k 2024-01-13T08:14:36.094 INFO:teuthology.orchestra.run.smithi099.stdout: skopeo x86_64 1:1.2.4-2.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 6.2 M 2024-01-13T08:14:36.094 INFO:teuthology.orchestra.run.smithi099.stdout: toolbox x86_64 0.0.99.3-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 2.2 M 2024-01-13T08:14:36.094 INFO:teuthology.orchestra.run.smithi099.stdout: udica noarch 0.2.4-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 51 k 2024-01-13T08:14:36.094 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-01-13T08:14:36.094 INFO:teuthology.orchestra.run.smithi099.stdout: yajl x86_64 2.1.0-12.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2024-01-13T08:14:36.094 INFO:teuthology.orchestra.run.smithi099.stdout:Downgrading: 2024-01-13T08:14:36.094 INFO:teuthology.orchestra.run.smithi099.stdout: conmon x86_64 2:2.0.26-3.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 51 k 2024-01-13T08:14:36.094 INFO:teuthology.orchestra.run.smithi099.stdout: container-selinux noarch 2:2.189.0-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 55 k 2024-01-13T08:14:36.094 INFO:teuthology.orchestra.run.smithi099.stdout: containernetworking-plugins x86_64 0.9.1-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 19 M 2024-01-13T08:14:36.094 INFO:teuthology.orchestra.run.smithi099.stdout: containers-common x86_64 1:1.2.4-2.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 91 k 2024-01-13T08:14:36.094 INFO:teuthology.orchestra.run.smithi099.stdout: criu x86_64 3.15-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 511 k 2024-01-13T08:14:36.095 INFO:teuthology.orchestra.run.smithi099.stdout: fuse-overlayfs x86_64 1.4.0-2.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 72 k 2024-01-13T08:14:36.095 INFO:teuthology.orchestra.run.smithi099.stdout: libslirp x86_64 4.3.1-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 69 k 2024-01-13T08:14:36.095 INFO:teuthology.orchestra.run.smithi099.stdout: podman x86_64 3.0.1-15.module+el8.7.0+18021+1e286cc8 rhel-8-for-x86_64-appstream-rpms 11 M 2024-01-13T08:14:36.095 INFO:teuthology.orchestra.run.smithi099.stdout: podman-catatonit x86_64 3.0.1-15.module+el8.7.0+18021+1e286cc8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-01-13T08:14:36.095 INFO:teuthology.orchestra.run.smithi099.stdout: podman-docker noarch 3.0.1-15.module+el8.7.0+18021+1e286cc8 rhel-8-for-x86_64-appstream-rpms 57 k 2024-01-13T08:14:36.095 INFO:teuthology.orchestra.run.smithi099.stdout: runc x86_64 1.0.0-73.rc95.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 2.9 M 2024-01-13T08:14:36.095 INFO:teuthology.orchestra.run.smithi099.stdout: slirp4netns x86_64 1.1.8-1.module+el8.7.0+16533+44634e24 rhel-8-for-x86_64-appstream-rpms 51 k 2024-01-13T08:14:36.095 INFO:teuthology.orchestra.run.smithi099.stdout:Installing module profiles: 2024-01-13T08:14:36.095 INFO:teuthology.orchestra.run.smithi099.stdout: container-tools/common 2024-01-13T08:14:36.095 INFO:teuthology.orchestra.run.smithi099.stdout:Enabling module streams: 2024-01-13T08:14:36.095 INFO:teuthology.orchestra.run.smithi099.stdout: container-tools 3.0 2024-01-13T08:14:36.095 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:14:36.095 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-01-13T08:14:36.095 INFO:teuthology.orchestra.run.smithi099.stdout:========================================================================================================================== 2024-01-13T08:14:36.095 INFO:teuthology.orchestra.run.smithi099.stdout:Install 7 Packages 2024-01-13T08:14:36.096 INFO:teuthology.orchestra.run.smithi099.stdout:Downgrade 12 Packages 2024-01-13T08:14:36.096 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:14:36.096 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 50 M 2024-01-13T08:14:36.096 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-01-13T08:14:36.262 INFO:teuthology.orchestra.run.smithi131.stdout:(1/19): criu-3.15-1.module+el8.7.0+16533+44634e 2.5 MB/s | 511 kB 00:00 2024-01-13T08:14:36.335 INFO:teuthology.orchestra.run.smithi099.stdout:(1/19): criu-3.15-1.module+el8.7.0+16533+44634e 2.1 MB/s | 511 kB 00:00 2024-01-13T08:14:36.439 INFO:teuthology.orchestra.run.smithi131.stdout:(2/19): conmon-2.0.26-3.module+el8.7.0+16533+44 136 kB/s | 51 kB 00:00 2024-01-13T08:14:36.469 INFO:teuthology.orchestra.run.smithi099.stdout:(2/19): fuse-overlayfs-1.4.0-2.module+el8.7.0+1 540 kB/s | 72 kB 00:00 2024-01-13T08:14:36.471 INFO:teuthology.orchestra.run.smithi131.stdout:(3/19): fuse-overlayfs-1.4.0-2.module+el8.7.0+1 345 kB/s | 72 kB 00:00 2024-01-13T08:14:36.585 INFO:teuthology.orchestra.run.smithi099.stdout:(3/19): conmon-2.0.26-3.module+el8.7.0+16533+44 105 kB/s | 51 kB 00:00 2024-01-13T08:14:36.680 INFO:teuthology.orchestra.run.smithi131.stdout:(4/19): containernetworking-plugins-0.9.1-1.mod 30 MB/s | 19 MB 00:00 2024-01-13T08:14:36.734 INFO:teuthology.orchestra.run.smithi131.stdout:(5/19): libslirp-4.3.1-1.module+el8.7.0+16533+4 234 kB/s | 69 kB 00:00 2024-01-13T08:14:36.752 INFO:teuthology.orchestra.run.smithi099.stdout:(4/19): libslirp-4.3.1-1.module+el8.7.0+16533+4 244 kB/s | 69 kB 00:00 2024-01-13T08:14:36.856 INFO:teuthology.orchestra.run.smithi131.stdout:(6/19): podman-docker-3.0.1-15.module+el8.7.0+1 326 kB/s | 57 kB 00:00 2024-01-13T08:14:36.894 INFO:teuthology.orchestra.run.smithi099.stdout:(5/19): podman-docker-3.0.1-15.module+el8.7.0+1 404 kB/s | 57 kB 00:00 2024-01-13T08:14:36.939 INFO:teuthology.orchestra.run.smithi131.stdout:(7/19): runc-1.0.0-73.rc95.module+el8.7.0+16533 14 MB/s | 2.9 MB 00:00 2024-01-13T08:14:37.073 INFO:teuthology.orchestra.run.smithi131.stdout:(8/19): podman-3.0.1-15.module+el8.7.0+18021+1e 18 MB/s | 11 MB 00:00 2024-01-13T08:14:37.086 INFO:teuthology.orchestra.run.smithi099.stdout:(6/19): containernetworking-plugins-0.9.1-1.mod 19 MB/s | 19 MB 00:00 2024-01-13T08:14:37.090 INFO:teuthology.orchestra.run.smithi131.stdout:(9/19): container-selinux-2.189.0-1.module+el8. 234 kB/s | 55 kB 00:00 2024-01-13T08:14:37.132 INFO:teuthology.orchestra.run.smithi131.stdout:(10/19): containers-common-1.2.4-2.module+el8.7 474 kB/s | 91 kB 00:00 2024-01-13T08:14:37.170 INFO:teuthology.orchestra.run.smithi099.stdout:(7/19): runc-1.0.0-73.rc95.module+el8.7.0+16533 10 MB/s | 2.9 MB 00:00 2024-01-13T08:14:37.332 INFO:teuthology.orchestra.run.smithi131.stdout:(11/19): slirp4netns-1.1.8-1.module+el8.7.0+165 212 kB/s | 51 kB 00:00 2024-01-13T08:14:37.337 INFO:teuthology.orchestra.run.smithi099.stdout:(8/19): podman-3.0.1-15.module+el8.7.0+18021+1e 14 MB/s | 11 MB 00:00 2024-01-13T08:14:37.362 INFO:teuthology.orchestra.run.smithi099.stdout:(9/19): container-selinux-2.189.0-1.module+el8. 199 kB/s | 55 kB 00:00 2024-01-13T08:14:37.516 INFO:teuthology.orchestra.run.smithi131.stdout:(12/19): crun-0.18-3.module+el8.7.0+16533+44634 1.0 MB/s | 185 kB 00:00 2024-01-13T08:14:37.529 INFO:teuthology.orchestra.run.smithi099.stdout:(10/19): containers-common-1.2.4-2.module+el8.7 254 kB/s | 91 kB 00:00 2024-01-13T08:14:37.566 INFO:teuthology.orchestra.run.smithi131.stdout:(13/19): podman-catatonit-3.0.1-15.module+el8.7 655 kB/s | 323 kB 00:00 2024-01-13T08:14:37.596 INFO:teuthology.orchestra.run.smithi099.stdout:(11/19): podman-catatonit-3.0.1-15.module+el8.7 1.2 MB/s | 323 kB 00:00 2024-01-13T08:14:37.704 INFO:teuthology.orchestra.run.smithi099.stdout:(12/19): slirp4netns-1.1.8-1.module+el8.7.0+165 150 kB/s | 51 kB 00:00 2024-01-13T08:14:37.766 INFO:teuthology.orchestra.run.smithi131.stdout:(14/19): cockpit-podman-29-2.module+el8.7.0+165 1.7 MB/s | 1.1 MB 00:00 2024-01-13T08:14:37.796 INFO:teuthology.orchestra.run.smithi099.stdout:(13/19): cockpit-podman-29-2.module+el8.7.0+165 4.0 MB/s | 1.1 MB 00:00 2024-01-13T08:14:37.842 INFO:teuthology.orchestra.run.smithi131.stdout:(15/19): toolbox-0.0.99.3-1.module+el8.7.0+1653 8.0 MB/s | 2.2 MB 00:00 2024-01-13T08:14:37.846 INFO:teuthology.orchestra.run.smithi099.stdout:(14/19): crun-0.18-3.module+el8.7.0+16533+44634 738 kB/s | 185 kB 00:00 2024-01-13T08:14:37.922 INFO:teuthology.orchestra.run.smithi099.stdout:(15/19): toolbox-0.0.99.3-1.module+el8.7.0+1653 18 MB/s | 2.2 MB 00:00 2024-01-13T08:14:37.972 INFO:teuthology.orchestra.run.smithi099.stdout:(16/19): udica-0.2.4-1.module+el8.7.0+16533+446 403 kB/s | 51 kB 00:00 2024-01-13T08:14:37.992 INFO:teuthology.orchestra.run.smithi131.stdout:(16/19): udica-0.2.4-1.module+el8.7.0+16533+446 225 kB/s | 51 kB 00:00 2024-01-13T08:14:38.081 INFO:teuthology.orchestra.run.smithi099.stdout:(17/19): skopeo-1.2.4-2.module+el8.7.0+16533+44 16 MB/s | 6.2 MB 00:00 2024-01-13T08:14:38.106 INFO:teuthology.orchestra.run.smithi099.stdout:(18/19): yajl-2.1.0-12.el8.x86_64.rpm 307 kB/s | 41 kB 00:00 2024-01-13T08:14:38.142 INFO:teuthology.orchestra.run.smithi131.stdout:(17/19): skopeo-1.2.4-2.module+el8.7.0+16533+44 9.9 MB/s | 6.2 MB 00:00 2024-01-13T08:14:38.167 INFO:teuthology.orchestra.run.smithi131.stdout:(18/19): yajl-2.1.0-12.el8.x86_64.rpm 233 kB/s | 41 kB 00:00 2024-01-13T08:14:38.248 INFO:teuthology.orchestra.run.smithi099.stdout:(19/19): buildah-1.19.9-6.module+el8.7.0+16533+ 20 MB/s | 6.6 MB 00:00 2024-01-13T08:14:38.249 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-01-13T08:14:38.249 INFO:teuthology.orchestra.run.smithi099.stdout:Total 23 MB/s | 50 MB 00:02 2024-01-13T08:14:38.296 INFO:teuthology.orchestra.run.smithi131.stdout:(19/19): buildah-1.19.9-6.module+el8.7.0+16533+ 15 MB/s | 6.6 MB 00:00 2024-01-13T08:14:38.296 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2024-01-13T08:14:38.296 INFO:teuthology.orchestra.run.smithi131.stdout:Total 22 MB/s | 50 MB 00:02 2024-01-13T08:14:38.726 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-01-13T08:14:38.770 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-01-13T08:14:38.771 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-01-13T08:14:38.782 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2024-01-13T08:14:38.826 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2024-01-13T08:14:38.826 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2024-01-13T08:14:39.200 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-01-13T08:14:39.208 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-01-13T08:14:39.249 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2024-01-13T08:14:39.258 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2024-01-13T08:14:40.346 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-01-13T08:14:40.361 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: container-selinux-2:2.189.0-1.module+el8.7.0+16533+4 1/1 2024-01-13T08:14:40.396 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: container-selinux-2:2.189.0-1.module+el8.7.0+16533 1/31 2024-01-13T08:14:40.419 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2024-01-13T08:14:40.435 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: container-selinux-2:2.189.0-1.module+el8.7.0+16533+4 1/1 2024-01-13T08:14:40.473 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: container-selinux-2:2.189.0-1.module+el8.7.0+16533 1/31 2024-01-13T08:14:40.596 INFO:teuthology.orchestra.run.smithi099.stdout: Downgrading : container-selinux-2:2.189.0-1.module+el8.7.0+16533 1/31 2024-01-13T08:14:40.625 INFO:teuthology.orchestra.run.smithi131.stdout: Downgrading : container-selinux-2:2.189.0-1.module+el8.7.0+16533 1/31 2024-01-13T08:14:57.284 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: container-selinux-2:2.189.0-1.module+el8.7.0+16533 1/31 2024-01-13T08:14:57.457 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: container-selinux-2:2.189.0-1.module+el8.7.0+16533 1/31 2024-01-13T08:14:57.484 INFO:teuthology.orchestra.run.smithi099.stdout: Downgrading : fuse-overlayfs-1.4.0-2.module+el8.7.0+16533+44634e 2/31 2024-01-13T08:14:57.501 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: fuse-overlayfs-1.4.0-2.module+el8.7.0+16533+44634e 2/31 2024-01-13T08:14:57.681 INFO:teuthology.orchestra.run.smithi131.stdout: Downgrading : fuse-overlayfs-1.4.0-2.module+el8.7.0+16533+44634e 2/31 2024-01-13T08:14:57.691 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : yajl-2.1.0-12.el8.x86_64 3/31 2024-01-13T08:14:57.699 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: fuse-overlayfs-1.4.0-2.module+el8.7.0+16533+44634e 2/31 2024-01-13T08:14:57.911 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : yajl-2.1.0-12.el8.x86_64 3/31 2024-01-13T08:14:57.914 INFO:teuthology.orchestra.run.smithi099.stdout: Downgrading : libslirp-4.3.1-1.module+el8.7.0+16533+44634e24.x86 4/31 2024-01-13T08:14:58.078 INFO:teuthology.orchestra.run.smithi131.stdout: Downgrading : libslirp-4.3.1-1.module+el8.7.0+16533+44634e24.x86 4/31 2024-01-13T08:14:58.125 INFO:teuthology.orchestra.run.smithi099.stdout: Downgrading : slirp4netns-1.1.8-1.module+el8.7.0+16533+44634e24. 5/31 2024-01-13T08:14:58.125 INFO:teuthology.orchestra.run.smithi099.stdout: Downgrading : containers-common-1:1.2.4-2.module+el8.7.0+16533+4 6/31 2024-01-13T08:14:58.125 INFO:teuthology.orchestra.run.smithi099.stdout:warning: /etc/containers/registries.conf created as /etc/containers/registries.conf.rpmnew 2024-01-13T08:14:58.125 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:14:58.298 INFO:teuthology.orchestra.run.smithi131.stdout: Downgrading : slirp4netns-1.1.8-1.module+el8.7.0+16533+44634e24. 5/31 2024-01-13T08:14:58.298 INFO:teuthology.orchestra.run.smithi131.stdout: Downgrading : containers-common-1:1.2.4-2.module+el8.7.0+16533+4 6/31 2024-01-13T08:14:58.299 INFO:teuthology.orchestra.run.smithi131.stdout:warning: /etc/containers/registries.conf created as /etc/containers/registries.conf.rpmnew 2024-01-13T08:14:58.299 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-01-13T08:14:58.920 INFO:teuthology.orchestra.run.smithi099.stdout: Downgrading : criu-3.15-1.module+el8.7.0+16533+44634e24.x86_64 7/31 2024-01-13T08:14:59.181 INFO:teuthology.orchestra.run.smithi131.stdout: Downgrading : criu-3.15-1.module+el8.7.0+16533+44634e24.x86_64 7/31 2024-01-13T08:15:00.669 INFO:teuthology.orchestra.run.smithi099.stdout: Downgrading : runc-1.0.0-73.rc95.module+el8.7.0+16533+44634e24.x 8/31 2024-01-13T08:15:00.898 INFO:teuthology.orchestra.run.smithi099.stdout: Downgrading : containernetworking-plugins-0.9.1-1.module+el8.7.0 9/31 2024-01-13T08:15:00.934 INFO:teuthology.orchestra.run.smithi131.stdout: Downgrading : runc-1.0.0-73.rc95.module+el8.7.0+16533+44634e24.x 8/31 2024-01-13T08:15:01.133 INFO:teuthology.orchestra.run.smithi099.stdout: Downgrading : conmon-2:2.0.26-3.module+el8.7.0+16533+44634e24.x8 10/31 2024-01-13T08:15:01.187 INFO:teuthology.orchestra.run.smithi131.stdout: Downgrading : containernetworking-plugins-0.9.1-1.module+el8.7.0 9/31 2024-01-13T08:15:01.423 INFO:teuthology.orchestra.run.smithi131.stdout: Downgrading : conmon-2:2.0.26-3.module+el8.7.0+16533+44634e24.x8 10/31 2024-01-13T08:15:02.526 INFO:teuthology.orchestra.run.smithi099.stdout: Downgrading : podman-catatonit-3.0.1-15.module+el8.7.0+18021+1e2 11/31 2024-01-13T08:15:02.835 INFO:teuthology.orchestra.run.smithi099.stdout: Downgrading : podman-3.0.1-15.module+el8.7.0+18021+1e286cc8.x86_ 12/31 2024-01-13T08:15:02.841 INFO:teuthology.orchestra.run.smithi131.stdout: Downgrading : podman-catatonit-3.0.1-15.module+el8.7.0+18021+1e2 11/31 2024-01-13T08:15:03.123 INFO:teuthology.orchestra.run.smithi099.stdout: Downgrading : podman-docker-3.0.1-15.module+el8.7.0+18021+1e286c 13/31 2024-01-13T08:15:03.125 INFO:teuthology.orchestra.run.smithi131.stdout: Downgrading : podman-3.0.1-15.module+el8.7.0+18021+1e286cc8.x86_ 12/31 2024-01-13T08:15:03.421 INFO:teuthology.orchestra.run.smithi131.stdout: Downgrading : podman-docker-3.0.1-15.module+el8.7.0+18021+1e286c 13/31 2024-01-13T08:15:03.489 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : cockpit-podman-29-2.module+el8.7.0+16533+44634e24. 14/31 2024-01-13T08:15:03.786 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : cockpit-podman-29-2.module+el8.7.0+16533+44634e24. 14/31 2024-01-13T08:15:04.239 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : toolbox-0.0.99.3-1.module+el8.7.0+16533+44634e24.x 15/31 2024-01-13T08:15:04.494 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : toolbox-0.0.99.3-1.module+el8.7.0+16533+44634e24.x 15/31 2024-01-13T08:15:04.953 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : buildah-1.19.9-6.module+el8.7.0+16533+44634e24.x86 16/31 2024-01-13T08:15:05.181 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : skopeo-1:1.2.4-2.module+el8.7.0+16533+44634e24.x86 17/31 2024-01-13T08:15:05.228 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : buildah-1.19.9-6.module+el8.7.0+16533+44634e24.x86 16/31 2024-01-13T08:15:05.431 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : crun-0.18-3.module+el8.7.0+16533+44634e24.x86_64 18/31 2024-01-13T08:15:05.454 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : skopeo-1:1.2.4-2.module+el8.7.0+16533+44634e24.x86 17/31 2024-01-13T08:15:05.647 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : udica-0.2.4-1.module+el8.7.0+16533+44634e24.noarch 19/31 2024-01-13T08:15:05.705 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : crun-0.18-3.module+el8.7.0+16533+44634e24.x86_64 18/31 2024-01-13T08:15:05.791 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : podman-docker-3:4.6.1-4.module+el8.9.0+20326+38708 20/31 2024-01-13T08:15:05.805 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: podman-3:4.6.1-4.module+el8.9.0+20326+387084d0.x86 21/31 2024-01-13T08:15:05.805 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : podman-3:4.6.1-4.module+el8.9.0+20326+387084d0.x86 21/31 2024-01-13T08:15:05.946 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : udica-0.2.4-1.module+el8.7.0+16533+44634e24.noarch 19/31 2024-01-13T08:15:06.106 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : podman-docker-3:4.6.1-4.module+el8.9.0+20326+38708 20/31 2024-01-13T08:15:06.118 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: podman-3:4.6.1-4.module+el8.9.0+20326+387084d0.x86 21/31 2024-01-13T08:15:06.119 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: podman-3:4.6.1-4.module+el8.9.0+20326+387084d0.x86 21/31 2024-01-13T08:15:06.120 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : podman-3:4.6.1-4.module+el8.9.0+20326+387084d0.x86 21/31 2024-01-13T08:15:06.300 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : containers-common-2:1-70.module+el8.9.0+20326+3870 22/31 2024-01-13T08:15:06.300 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : container-selinux-2:2.221.0-1.module+el8.9.0+20326 23/31 2024-01-13T08:15:06.444 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: podman-3:4.6.1-4.module+el8.9.0+20326+387084d0.x86 21/31 2024-01-13T08:15:06.517 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: container-selinux-2:2.221.0-1.module+el8.9.0+20326 23/31 2024-01-13T08:15:06.517 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : podman-catatonit-3:4.6.1-4.module+el8.9.0+20326+38 24/31 2024-01-13T08:15:06.668 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : containers-common-2:1-70.module+el8.9.0+20326+3870 22/31 2024-01-13T08:15:06.668 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : container-selinux-2:2.221.0-1.module+el8.9.0+20326 23/31 2024-01-13T08:15:06.675 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: podman-catatonit-3:4.6.1-4.module+el8.9.0+20326+38 24/31 2024-01-13T08:15:06.866 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : slirp4netns-1.2.1-1.module+el8.9.0+20326+387084d0. 25/31 2024-01-13T08:15:06.882 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: container-selinux-2:2.221.0-1.module+el8.9.0+20326 23/31 2024-01-13T08:15:06.882 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : podman-catatonit-3:4.6.1-4.module+el8.9.0+20326+38 24/31 2024-01-13T08:15:07.031 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: podman-catatonit-3:4.6.1-4.module+el8.9.0+20326+38 24/31 2024-01-13T08:15:07.099 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : runc-1:1.1.9-1.module+el8.9.0+20326+387084d0.x86_6 26/31 2024-01-13T08:15:07.214 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : slirp4netns-1.2.1-1.module+el8.9.0+20326+387084d0. 25/31 2024-01-13T08:15:07.292 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : criu-3.18-4.module+el8.9.0+20326+387084d0.x86_64 27/31 2024-01-13T08:15:07.364 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : runc-1:1.1.9-1.module+el8.9.0+20326+387084d0.x86_6 26/31 2024-01-13T08:15:07.474 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : libslirp-4.4.0-1.module+el8.9.0+20326+387084d0.x86 28/31 2024-01-13T08:15:07.585 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : criu-3.18-4.module+el8.9.0+20326+387084d0.x86_64 27/31 2024-01-13T08:15:07.674 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : fuse-overlayfs-1.12-1.module+el8.9.0+20326+387084d 29/31 2024-01-13T08:15:07.748 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : libslirp-4.4.0-1.module+el8.9.0+20326+387084d0.x86 28/31 2024-01-13T08:15:07.877 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : conmon-3:2.1.8-1.module+el8.9.0+20326+387084d0.x86 30/31 2024-01-13T08:15:07.878 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : containernetworking-plugins-1:1.3.0-4.module+el8.9 31/31 2024-01-13T08:15:07.905 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : fuse-overlayfs-1.12-1.module+el8.9.0+20326+387084d 29/31 2024-01-13T08:15:08.091 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: containernetworking-plugins-1:1.3.0-4.module+el8.9 31/31 2024-01-13T08:15:08.091 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : conmon-3:2.1.8-1.module+el8.9.0+20326+387084d0.x86 30/31 2024-01-13T08:15:08.092 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : containernetworking-plugins-1:1.3.0-4.module+el8.9 31/31 2024-01-13T08:15:08.272 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: containernetworking-plugins-1:1.3.0-4.module+el8.9 31/31 2024-01-13T08:15:25.734 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: container-selinux-2:2.189.0-1.module+el8.7.0+16533 31/31 2024-01-13T08:15:25.980 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: container-selinux-2:2.189.0-1.module+el8.7.0+16533 31/31 2024-01-13T08:15:26.166 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: containernetworking-plugins-1:1.3.0-4.module+el8.9 31/31 2024-01-13T08:15:26.166 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : conmon-2:2.0.26-3.module+el8.7.0+16533+44634e24.x8 1/31 2024-01-13T08:15:26.166 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : conmon-3:2.1.8-1.module+el8.9.0+20326+387084d0.x86 2/31 2024-01-13T08:15:26.166 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : containernetworking-plugins-0.9.1-1.module+el8.7.0 3/31 2024-01-13T08:15:26.166 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : containernetworking-plugins-1:1.3.0-4.module+el8.9 4/31 2024-01-13T08:15:26.166 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : criu-3.15-1.module+el8.7.0+16533+44634e24.x86_64 5/31 2024-01-13T08:15:26.166 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : criu-3.18-4.module+el8.9.0+20326+387084d0.x86_64 6/31 2024-01-13T08:15:26.166 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : fuse-overlayfs-1.4.0-2.module+el8.7.0+16533+44634e 7/31 2024-01-13T08:15:26.166 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : fuse-overlayfs-1.12-1.module+el8.9.0+20326+387084d 8/31 2024-01-13T08:15:26.166 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libslirp-4.3.1-1.module+el8.7.0+16533+44634e24.x86 9/31 2024-01-13T08:15:26.166 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libslirp-4.4.0-1.module+el8.9.0+20326+387084d0.x86 10/31 2024-01-13T08:15:26.166 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : podman-3.0.1-15.module+el8.7.0+18021+1e286cc8.x86_ 11/31 2024-01-13T08:15:26.166 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : podman-3:4.6.1-4.module+el8.9.0+20326+387084d0.x86 12/31 2024-01-13T08:15:26.167 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : podman-docker-3.0.1-15.module+el8.7.0+18021+1e286c 13/31 2024-01-13T08:15:26.167 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : podman-docker-3:4.6.1-4.module+el8.9.0+20326+38708 14/31 2024-01-13T08:15:26.167 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : runc-1.0.0-73.rc95.module+el8.7.0+16533+44634e24.x 15/31 2024-01-13T08:15:26.167 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : runc-1:1.1.9-1.module+el8.9.0+20326+387084d0.x86_6 16/31 2024-01-13T08:15:26.167 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : container-selinux-2:2.189.0-1.module+el8.7.0+16533 17/31 2024-01-13T08:15:26.167 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : container-selinux-2:2.221.0-1.module+el8.9.0+20326 18/31 2024-01-13T08:15:26.167 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : containers-common-1:1.2.4-2.module+el8.7.0+16533+4 19/31 2024-01-13T08:15:26.167 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : containers-common-2:1-70.module+el8.9.0+20326+3870 20/31 2024-01-13T08:15:26.167 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : podman-catatonit-3.0.1-15.module+el8.7.0+18021+1e2 21/31 2024-01-13T08:15:26.167 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : podman-catatonit-3:4.6.1-4.module+el8.9.0+20326+38 22/31 2024-01-13T08:15:26.167 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : slirp4netns-1.1.8-1.module+el8.7.0+16533+44634e24. 23/31 2024-01-13T08:15:26.167 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : slirp4netns-1.2.1-1.module+el8.9.0+20326+387084d0. 24/31 2024-01-13T08:15:26.167 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : cockpit-podman-29-2.module+el8.7.0+16533+44634e24. 25/31 2024-01-13T08:15:26.167 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : crun-0.18-3.module+el8.7.0+16533+44634e24.x86_64 26/31 2024-01-13T08:15:26.167 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : skopeo-1:1.2.4-2.module+el8.7.0+16533+44634e24.x86 27/31 2024-01-13T08:15:26.168 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : toolbox-0.0.99.3-1.module+el8.7.0+16533+44634e24.x 28/31 2024-01-13T08:15:26.168 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : udica-0.2.4-1.module+el8.7.0+16533+44634e24.noarch 29/31 2024-01-13T08:15:26.168 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : buildah-1.19.9-6.module+el8.7.0+16533+44634e24.x86 30/31 2024-01-13T08:15:26.473 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: containernetworking-plugins-1:1.3.0-4.module+el8.9 31/31 2024-01-13T08:15:26.473 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : conmon-2:2.0.26-3.module+el8.7.0+16533+44634e24.x8 1/31 2024-01-13T08:15:26.473 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : conmon-3:2.1.8-1.module+el8.9.0+20326+387084d0.x86 2/31 2024-01-13T08:15:26.473 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : containernetworking-plugins-0.9.1-1.module+el8.7.0 3/31 2024-01-13T08:15:26.473 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : containernetworking-plugins-1:1.3.0-4.module+el8.9 4/31 2024-01-13T08:15:26.473 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : criu-3.15-1.module+el8.7.0+16533+44634e24.x86_64 5/31 2024-01-13T08:15:26.473 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : criu-3.18-4.module+el8.9.0+20326+387084d0.x86_64 6/31 2024-01-13T08:15:26.473 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : fuse-overlayfs-1.4.0-2.module+el8.7.0+16533+44634e 7/31 2024-01-13T08:15:26.473 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : fuse-overlayfs-1.12-1.module+el8.9.0+20326+387084d 8/31 2024-01-13T08:15:26.473 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libslirp-4.3.1-1.module+el8.7.0+16533+44634e24.x86 9/31 2024-01-13T08:15:26.473 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libslirp-4.4.0-1.module+el8.9.0+20326+387084d0.x86 10/31 2024-01-13T08:15:26.473 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : podman-3.0.1-15.module+el8.7.0+18021+1e286cc8.x86_ 11/31 2024-01-13T08:15:26.474 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : podman-3:4.6.1-4.module+el8.9.0+20326+387084d0.x86 12/31 2024-01-13T08:15:26.474 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : podman-docker-3.0.1-15.module+el8.7.0+18021+1e286c 13/31 2024-01-13T08:15:26.474 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : podman-docker-3:4.6.1-4.module+el8.9.0+20326+38708 14/31 2024-01-13T08:15:26.474 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : runc-1.0.0-73.rc95.module+el8.7.0+16533+44634e24.x 15/31 2024-01-13T08:15:26.474 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : runc-1:1.1.9-1.module+el8.9.0+20326+387084d0.x86_6 16/31 2024-01-13T08:15:26.474 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : container-selinux-2:2.189.0-1.module+el8.7.0+16533 17/31 2024-01-13T08:15:26.474 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : container-selinux-2:2.221.0-1.module+el8.9.0+20326 18/31 2024-01-13T08:15:26.474 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : containers-common-1:1.2.4-2.module+el8.7.0+16533+4 19/31 2024-01-13T08:15:26.474 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : containers-common-2:1-70.module+el8.9.0+20326+3870 20/31 2024-01-13T08:15:26.474 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : podman-catatonit-3.0.1-15.module+el8.7.0+18021+1e2 21/31 2024-01-13T08:15:26.474 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : podman-catatonit-3:4.6.1-4.module+el8.9.0+20326+38 22/31 2024-01-13T08:15:26.474 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : slirp4netns-1.1.8-1.module+el8.7.0+16533+44634e24. 23/31 2024-01-13T08:15:26.474 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : slirp4netns-1.2.1-1.module+el8.9.0+20326+387084d0. 24/31 2024-01-13T08:15:26.474 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : cockpit-podman-29-2.module+el8.7.0+16533+44634e24. 25/31 2024-01-13T08:15:26.475 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : crun-0.18-3.module+el8.7.0+16533+44634e24.x86_64 26/31 2024-01-13T08:15:26.475 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : skopeo-1:1.2.4-2.module+el8.7.0+16533+44634e24.x86 27/31 2024-01-13T08:15:26.475 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : toolbox-0.0.99.3-1.module+el8.7.0+16533+44634e24.x 28/31 2024-01-13T08:15:26.475 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : udica-0.2.4-1.module+el8.7.0+16533+44634e24.noarch 29/31 2024-01-13T08:15:26.475 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : buildah-1.19.9-6.module+el8.7.0+16533+44634e24.x86 30/31 2024-01-13T08:15:38.292 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 31/31 2024-01-13T08:15:38.293 INFO:teuthology.orchestra.run.smithi131.stdout:Installed products updated. 2024-01-13T08:15:38.425 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 31/31 2024-01-13T08:15:38.425 INFO:teuthology.orchestra.run.smithi099.stdout:Installed products updated. 2024-01-13T08:15:38.821 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-01-13T08:15:38.821 INFO:teuthology.orchestra.run.smithi131.stdout:Downgraded: 2024-01-13T08:15:38.821 INFO:teuthology.orchestra.run.smithi131.stdout: conmon-2:2.0.26-3.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T08:15:38.821 INFO:teuthology.orchestra.run.smithi131.stdout: container-selinux-2:2.189.0-1.module+el8.7.0+16533+44634e24.noarch 2024-01-13T08:15:38.821 INFO:teuthology.orchestra.run.smithi131.stdout: containernetworking-plugins-0.9.1-1.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T08:15:38.821 INFO:teuthology.orchestra.run.smithi131.stdout: containers-common-1:1.2.4-2.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T08:15:38.821 INFO:teuthology.orchestra.run.smithi131.stdout: criu-3.15-1.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T08:15:38.821 INFO:teuthology.orchestra.run.smithi131.stdout: fuse-overlayfs-1.4.0-2.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T08:15:38.821 INFO:teuthology.orchestra.run.smithi131.stdout: libslirp-4.3.1-1.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T08:15:38.822 INFO:teuthology.orchestra.run.smithi131.stdout: podman-3.0.1-15.module+el8.7.0+18021+1e286cc8.x86_64 2024-01-13T08:15:38.822 INFO:teuthology.orchestra.run.smithi131.stdout: podman-catatonit-3.0.1-15.module+el8.7.0+18021+1e286cc8.x86_64 2024-01-13T08:15:38.822 INFO:teuthology.orchestra.run.smithi131.stdout: podman-docker-3.0.1-15.module+el8.7.0+18021+1e286cc8.noarch 2024-01-13T08:15:38.822 INFO:teuthology.orchestra.run.smithi131.stdout: runc-1.0.0-73.rc95.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T08:15:38.822 INFO:teuthology.orchestra.run.smithi131.stdout: slirp4netns-1.1.8-1.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T08:15:38.822 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2024-01-13T08:15:38.822 INFO:teuthology.orchestra.run.smithi131.stdout: buildah-1.19.9-6.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T08:15:38.822 INFO:teuthology.orchestra.run.smithi131.stdout: cockpit-podman-29-2.module+el8.7.0+16533+44634e24.noarch 2024-01-13T08:15:38.822 INFO:teuthology.orchestra.run.smithi131.stdout: crun-0.18-3.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T08:15:38.822 INFO:teuthology.orchestra.run.smithi131.stdout: skopeo-1:1.2.4-2.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T08:15:38.822 INFO:teuthology.orchestra.run.smithi131.stdout: toolbox-0.0.99.3-1.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T08:15:38.822 INFO:teuthology.orchestra.run.smithi131.stdout: udica-0.2.4-1.module+el8.7.0+16533+44634e24.noarch 2024-01-13T08:15:38.822 INFO:teuthology.orchestra.run.smithi131.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-13T08:15:38.822 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-01-13T08:15:38.823 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2024-01-13T08:15:39.006 DEBUG:teuthology.parallel:result is None 2024-01-13T08:15:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:15:39.067 INFO:teuthology.orchestra.run.smithi099.stdout:Downgraded: 2024-01-13T08:15:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: conmon-2:2.0.26-3.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T08:15:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: container-selinux-2:2.189.0-1.module+el8.7.0+16533+44634e24.noarch 2024-01-13T08:15:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: containernetworking-plugins-0.9.1-1.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T08:15:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: containers-common-1:1.2.4-2.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T08:15:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: criu-3.15-1.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T08:15:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: fuse-overlayfs-1.4.0-2.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T08:15:39.067 INFO:teuthology.orchestra.run.smithi099.stdout: libslirp-4.3.1-1.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T08:15:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: podman-3.0.1-15.module+el8.7.0+18021+1e286cc8.x86_64 2024-01-13T08:15:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: podman-catatonit-3.0.1-15.module+el8.7.0+18021+1e286cc8.x86_64 2024-01-13T08:15:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: podman-docker-3.0.1-15.module+el8.7.0+18021+1e286cc8.noarch 2024-01-13T08:15:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: runc-1.0.0-73.rc95.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T08:15:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: slirp4netns-1.1.8-1.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T08:15:39.068 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-01-13T08:15:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: buildah-1.19.9-6.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T08:15:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: cockpit-podman-29-2.module+el8.7.0+16533+44634e24.noarch 2024-01-13T08:15:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: crun-0.18-3.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T08:15:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: skopeo-1:1.2.4-2.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T08:15:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: toolbox-0.0.99.3-1.module+el8.7.0+16533+44634e24.x86_64 2024-01-13T08:15:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: udica-0.2.4-1.module+el8.7.0+16533+44634e24.noarch 2024-01-13T08:15:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-13T08:15:39.068 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:15:39.069 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-01-13T08:15:39.243 DEBUG:teuthology.parallel:result is None 2024-01-13T08:15:39.244 INFO:teuthology.run_tasks:Running task nvme_loop... 2024-01-13T08:15:39.255 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2024-01-13T08:15:39.255 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-01-13T08:15:39.255 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/scratch_devs of=/dev/stdout 2024-01-13T08:15:39.272 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-13T08:15:39.273 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_1 2024-01-13T08:15:39.330 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-13T08:15:39.330 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-13T08:15:39.330 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 168439 Links: 1 2024-01-13T08:15:39.330 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-13T08:15:39.330 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2024-01-13T08:15:39.330 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-01-13 08:15:25.886523879 +0000 2024-01-13T08:15:39.330 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-01-13 08:13:36.899877646 +0000 2024-01-13T08:15:39.331 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-01-13 08:13:36.899877646 +0000 2024-01-13T08:15:39.331 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2024-01-13T08:15:39.331 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-13T08:15:39.397 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-01-13T08:15:39.397 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-01-13T08:15:39.397 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000298156 s, 1.7 MB/s 2024-01-13T08:15:39.398 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-13T08:15:39.457 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_2 2024-01-13T08:15:39.515 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-13T08:15:39.515 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-13T08:15:39.515 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 165793 Links: 1 2024-01-13T08:15:39.515 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-13T08:15:39.515 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2024-01-13T08:15:39.515 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-01-13 08:15:25.886523879 +0000 2024-01-13T08:15:39.515 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-01-13 08:13:37.253869404 +0000 2024-01-13T08:15:39.515 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-01-13 08:13:37.253869404 +0000 2024-01-13T08:15:39.515 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2024-01-13T08:15:39.516 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-13T08:15:39.580 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-01-13T08:15:39.581 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-01-13T08:15:39.581 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000201991 s, 2.5 MB/s 2024-01-13T08:15:39.582 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-13T08:15:39.639 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_3 2024-01-13T08:15:39.696 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-13T08:15:39.696 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-13T08:15:39.696 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 169884 Links: 1 2024-01-13T08:15:39.696 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-13T08:15:39.696 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2024-01-13T08:15:39.696 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-01-13 08:15:25.886523879 +0000 2024-01-13T08:15:39.696 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-01-13 08:13:37.598861372 +0000 2024-01-13T08:15:39.696 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-01-13 08:13:37.598861372 +0000 2024-01-13T08:15:39.696 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2024-01-13T08:15:39.697 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-13T08:15:39.760 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-01-13T08:15:39.760 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-01-13T08:15:39.760 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000209067 s, 2.4 MB/s 2024-01-13T08:15:39.761 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-13T08:15:39.819 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_4 2024-01-13T08:15:39.877 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-13T08:15:39.877 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-13T08:15:39.877 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 165883 Links: 1 2024-01-13T08:15:39.877 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-13T08:15:39.877 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2024-01-13T08:15:39.877 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-01-13 08:15:25.887523856 +0000 2024-01-13T08:15:39.877 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-01-13 08:13:37.923853806 +0000 2024-01-13T08:15:39.877 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-01-13 08:13:37.923853806 +0000 2024-01-13T08:15:39.877 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2024-01-13T08:15:39.878 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-13T08:15:39.942 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-01-13T08:15:39.942 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-01-13T08:15:39.942 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000210102 s, 2.4 MB/s 2024-01-13T08:15:39.943 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-13T08:15:40.001 DEBUG:teuthology.orchestra.run.smithi099:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-01-13T08:15:40.112 INFO:teuthology.orchestra.run.smithi099.stdout:loop 2024-01-13T08:15:40.113 INFO:tasks.nvme_loop:Connecting nvme_loop smithi099:/dev/vg_nvme/lv_1... 2024-01-13T08:15:40.113 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-01-13T08:15:40.149 INFO:teuthology.orchestra.run.smithi099.stdout:1 2024-01-13T08:15:40.184 INFO:teuthology.orchestra.run.smithi099.stdout:/dev/vg_nvme/lv_11 2024-01-13T08:15:40.224 INFO:tasks.nvme_loop:Connecting nvme_loop smithi099:/dev/vg_nvme/lv_2... 2024-01-13T08:15:40.225 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-01-13T08:15:40.300 INFO:teuthology.orchestra.run.smithi099.stdout:1 2024-01-13T08:15:40.335 INFO:teuthology.orchestra.run.smithi099.stdout:/dev/vg_nvme/lv_21 2024-01-13T08:15:40.374 INFO:tasks.nvme_loop:Connecting nvme_loop smithi099:/dev/vg_nvme/lv_3... 2024-01-13T08:15:40.374 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-01-13T08:15:40.449 INFO:teuthology.orchestra.run.smithi099.stdout:1 2024-01-13T08:15:40.484 INFO:teuthology.orchestra.run.smithi099.stdout:/dev/vg_nvme/lv_31 2024-01-13T08:15:40.523 INFO:tasks.nvme_loop:Connecting nvme_loop smithi099:/dev/vg_nvme/lv_4... 2024-01-13T08:15:40.523 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-01-13T08:15:40.599 INFO:teuthology.orchestra.run.smithi099.stdout:1 2024-01-13T08:15:40.634 INFO:teuthology.orchestra.run.smithi099.stdout:/dev/vg_nvme/lv_41 2024-01-13T08:15:40.673 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-01-13T08:15:40.673 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/scratch_devs of=/dev/stdout 2024-01-13T08:15:40.730 DEBUG:teuthology.orchestra.run.smithi099:> sudo nvme list 2024-01-13T08:15:40.795 INFO:teuthology.orchestra.run.smithi099.stdout:Node SN Model Namespace Usage Format FW Rev 2024-01-13T08:15:40.795 INFO:teuthology.orchestra.run.smithi099.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-01-13T08:15:40.796 INFO:teuthology.orchestra.run.smithi099.stdout:/dev/nvme0n1 PHFT62040101400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-01-13T08:15:40.796 INFO:teuthology.orchestra.run.smithi099.stdout:/dev/nvme1n1 e8579fd499f87200f832 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2024-01-13T08:15:40.796 INFO:teuthology.orchestra.run.smithi099.stdout:/dev/nvme2n1 8e6290c0528a8b470f51 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2024-01-13T08:15:40.796 INFO:teuthology.orchestra.run.smithi099.stdout:/dev/nvme3n1 1ecd6f60fca2eae8f46d Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2024-01-13T08:15:40.796 INFO:teuthology.orchestra.run.smithi099.stdout:/dev/nvme4n1 4009ce0efc4eecdf76e8 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2024-01-13T08:15:40.797 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-01-13T08:15:40.797 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-01-13T08:15:40.797 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/scratch_devs 2024-01-13T08:15:40.864 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-01-13T08:15:40.864 DEBUG:teuthology.orchestra.run.smithi131:> dd if=/scratch_devs of=/dev/stdout 2024-01-13T08:15:40.886 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-13T08:15:40.886 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/vg_nvme/lv_1 2024-01-13T08:15:40.946 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-13T08:15:40.946 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-13T08:15:40.946 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 6h/6d Inode: 165244 Links: 1 2024-01-13T08:15:40.946 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-13T08:15:40.946 INFO:teuthology.orchestra.run.smithi131.stdout:Context: system_u:object_r:device_t:s0 2024-01-13T08:15:40.946 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2024-01-13 08:15:26.188850016 +0000 2024-01-13T08:15:40.946 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2024-01-13 08:13:43.276156924 +0000 2024-01-13T08:15:40.946 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2024-01-13 08:13:43.276156924 +0000 2024-01-13T08:15:40.947 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: - 2024-01-13T08:15:40.947 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-13T08:15:41.016 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2024-01-13T08:15:41.016 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2024-01-13T08:15:41.017 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000258778 s, 2.0 MB/s 2024-01-13T08:15:41.018 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-13T08:15:41.077 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/vg_nvme/lv_2 2024-01-13T08:15:41.136 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-13T08:15:41.136 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-13T08:15:41.137 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 6h/6d Inode: 165291 Links: 1 2024-01-13T08:15:41.137 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-13T08:15:41.137 INFO:teuthology.orchestra.run.smithi131.stdout:Context: system_u:object_r:device_t:s0 2024-01-13T08:15:41.137 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2024-01-13 08:15:26.188850016 +0000 2024-01-13T08:15:41.137 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2024-01-13 08:13:43.629149425 +0000 2024-01-13T08:15:41.137 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2024-01-13 08:13:43.629149425 +0000 2024-01-13T08:15:41.137 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: - 2024-01-13T08:15:41.137 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-13T08:15:41.203 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2024-01-13T08:15:41.204 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2024-01-13T08:15:41.204 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000208531 s, 2.5 MB/s 2024-01-13T08:15:41.205 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-13T08:15:41.264 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/vg_nvme/lv_3 2024-01-13T08:15:41.322 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-13T08:15:41.322 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-13T08:15:41.322 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 6h/6d Inode: 173104 Links: 1 2024-01-13T08:15:41.322 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-13T08:15:41.323 INFO:teuthology.orchestra.run.smithi131.stdout:Context: system_u:object_r:device_t:s0 2024-01-13T08:15:41.323 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2024-01-13 08:15:26.189849995 +0000 2024-01-13T08:15:41.323 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2024-01-13 08:13:43.961142372 +0000 2024-01-13T08:15:41.323 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2024-01-13 08:13:43.961142372 +0000 2024-01-13T08:15:41.323 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: - 2024-01-13T08:15:41.323 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-13T08:15:41.390 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2024-01-13T08:15:41.390 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2024-01-13T08:15:41.390 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000223358 s, 2.3 MB/s 2024-01-13T08:15:41.391 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-13T08:15:41.451 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/vg_nvme/lv_4 2024-01-13T08:15:41.508 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-13T08:15:41.509 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-13T08:15:41.509 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 6h/6d Inode: 165437 Links: 1 2024-01-13T08:15:41.509 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-13T08:15:41.509 INFO:teuthology.orchestra.run.smithi131.stdout:Context: system_u:object_r:device_t:s0 2024-01-13T08:15:41.509 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2024-01-13 08:15:26.189849995 +0000 2024-01-13T08:15:41.509 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2024-01-13 08:13:44.303135107 +0000 2024-01-13T08:15:41.509 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2024-01-13 08:13:44.303135107 +0000 2024-01-13T08:15:41.509 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: - 2024-01-13T08:15:41.509 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-13T08:15:41.576 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2024-01-13T08:15:41.576 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2024-01-13T08:15:41.576 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000285235 s, 1.8 MB/s 2024-01-13T08:15:41.577 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-13T08:15:41.636 DEBUG:teuthology.orchestra.run.smithi131:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-01-13T08:15:41.774 INFO:teuthology.orchestra.run.smithi131.stdout:loop 2024-01-13T08:15:41.775 INFO:tasks.nvme_loop:Connecting nvme_loop smithi131:/dev/vg_nvme/lv_1... 2024-01-13T08:15:41.776 DEBUG:teuthology.orchestra.run.smithi131:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-01-13T08:15:41.811 INFO:teuthology.orchestra.run.smithi131.stdout:1 2024-01-13T08:15:41.846 INFO:teuthology.orchestra.run.smithi131.stdout:/dev/vg_nvme/lv_11 2024-01-13T08:15:41.886 INFO:tasks.nvme_loop:Connecting nvme_loop smithi131:/dev/vg_nvme/lv_2... 2024-01-13T08:15:41.887 DEBUG:teuthology.orchestra.run.smithi131:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-01-13T08:15:41.963 INFO:teuthology.orchestra.run.smithi131.stdout:1 2024-01-13T08:15:41.998 INFO:teuthology.orchestra.run.smithi131.stdout:/dev/vg_nvme/lv_21 2024-01-13T08:15:42.038 INFO:tasks.nvme_loop:Connecting nvme_loop smithi131:/dev/vg_nvme/lv_3... 2024-01-13T08:15:42.038 DEBUG:teuthology.orchestra.run.smithi131:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-01-13T08:15:42.113 INFO:teuthology.orchestra.run.smithi131.stdout:1 2024-01-13T08:15:42.148 INFO:teuthology.orchestra.run.smithi131.stdout:/dev/vg_nvme/lv_31 2024-01-13T08:15:42.187 INFO:tasks.nvme_loop:Connecting nvme_loop smithi131:/dev/vg_nvme/lv_4... 2024-01-13T08:15:42.187 DEBUG:teuthology.orchestra.run.smithi131:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-01-13T08:15:42.262 INFO:teuthology.orchestra.run.smithi131.stdout:1 2024-01-13T08:15:42.297 INFO:teuthology.orchestra.run.smithi131.stdout:/dev/vg_nvme/lv_41 2024-01-13T08:15:42.336 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-01-13T08:15:42.336 DEBUG:teuthology.orchestra.run.smithi131:> dd if=/scratch_devs of=/dev/stdout 2024-01-13T08:15:42.393 DEBUG:teuthology.orchestra.run.smithi131:> sudo nvme list 2024-01-13T08:15:42.458 INFO:teuthology.orchestra.run.smithi131.stdout:Node SN Model Namespace Usage Format FW Rev 2024-01-13T08:15:42.458 INFO:teuthology.orchestra.run.smithi131.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-01-13T08:15:42.458 INFO:teuthology.orchestra.run.smithi131.stdout:/dev/nvme0n1 CVFT623300N4400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-01-13T08:15:42.458 INFO:teuthology.orchestra.run.smithi131.stdout:/dev/nvme1n1 efb5cb3bb17d90001ba5 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2024-01-13T08:15:42.458 INFO:teuthology.orchestra.run.smithi131.stdout:/dev/nvme2n1 2783aef7a3b76990bc4f Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2024-01-13T08:15:42.458 INFO:teuthology.orchestra.run.smithi131.stdout:/dev/nvme3n1 6eb40d70ad50b231bd0a Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2024-01-13T08:15:42.458 INFO:teuthology.orchestra.run.smithi131.stdout:/dev/nvme4n1 837ed6582963f7982650 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 4.18.0-5 2024-01-13T08:15:42.459 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-01-13T08:15:42.459 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-01-13T08:15:42.459 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/scratch_devs 2024-01-13T08:15:42.527 INFO:teuthology.run_tasks:Running task cephadm... 2024-01-13T08:15:42.602 INFO:tasks.cephadm:Config: {'roleless': True, 'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '445562ab4bc3ddfb386936119050695810860bcb'} 2024-01-13T08:15:42.602 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb 2024-01-13T08:15:42.602 INFO:tasks.cephadm:Cluster fsid is f1ba87a2-b1eb-11ee-95ac-87774f69a715 2024-01-13T08:15:42.603 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-01-13T08:15:42.603 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-01-13T08:15:42.603 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi099': '172.21.15.99', 'mon.smithi131': '172.21.15.131'} 2024-01-13T08:15:42.603 INFO:tasks.cephadm:Normalizing hostnames... 2024-01-13T08:15:42.603 DEBUG:teuthology.orchestra.run.smithi099:> sudo hostname $(hostname -s) 2024-01-13T08:15:42.636 DEBUG:teuthology.orchestra.run.smithi131:> sudo hostname $(hostname -s) 2024-01-13T08:15:42.664 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph-ci.git ref 445562ab4bc3ddfb386936119050695810860bcb)... 2024-01-13T08:15:42.664 DEBUG:teuthology.orchestra.run.smithi099:> curl --silent https://raw.githubusercontent.com/ceph/ceph-ci/445562ab4bc3ddfb386936119050695810860bcb/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-13T08:15:42.928 INFO:teuthology.orchestra.run.smithi099.stdout:-rw-rw-r--. 1 ubuntu ubuntu 355729 Jan 13 08:15 /home/ubuntu/cephtest/cephadm 2024-01-13T08:15:42.928 DEBUG:teuthology.orchestra.run.smithi131:> curl --silent https://raw.githubusercontent.com/ceph/ceph-ci/445562ab4bc3ddfb386936119050695810860bcb/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-13T08:15:43.111 INFO:teuthology.orchestra.run.smithi131.stdout:-rw-rw-r--. 1 ubuntu ubuntu 355729 Jan 13 08:15 /home/ubuntu/cephtest/cephadm 2024-01-13T08:15:43.112 DEBUG:teuthology.orchestra.run.smithi099:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-13T08:15:43.133 DEBUG:teuthology.orchestra.run.smithi131:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-13T08:15:43.161 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /etc/ceph 2024-01-13T08:15:43.206 DEBUG:teuthology.orchestra.run.smithi131:> sudo mkdir -p /etc/ceph 2024-01-13T08:15:43.233 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 777 /etc/ceph 2024-01-13T08:15:43.278 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod 777 /etc/ceph 2024-01-13T08:15:43.304 INFO:tasks.cephadm:Writing seed config... 2024-01-13T08:15:43.305 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-01-13T08:15:43.306 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-01-13T08:15:43.306 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-01-13T08:15:43.306 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-01-13T08:15:43.306 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-01-13T08:15:43.306 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-01-13T08:15:43.306 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-01-13T08:15:43.306 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-01-13T08:15:43.307 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-01-13T08:15:43.307 DEBUG:teuthology.orchestra.run.smithi099:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-01-13T08:15:43.337 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = f1ba87a2-b1eb-11ee-95ac-87774f69a715 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-01-13T08:15:43.338 DEBUG:teuthology.orchestra.run.smithi099:mon.smithi099> sudo journalctl -f -n 0 -u ceph-f1ba87a2-b1eb-11ee-95ac-87774f69a715@mon.smithi099.service 2024-01-13T08:15:43.381 INFO:tasks.cephadm:Bootstrapping... 2024-01-13T08:15:43.381 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb -v bootstrap --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.99 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-01-13T08:15:43.409 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:-- Logs begin at Sat 2024-01-13 08:05:08 UTC. -- 2024-01-13T08:15:43.676 INFO:teuthology.orchestra.run.smithi099.stderr:-------------------------------------------------------------------------------- 2024-01-13T08:15:43.676 INFO:teuthology.orchestra.run.smithi099.stderr:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb', '-v', 'bootstrap', '--fsid', 'f1ba87a2-b1eb-11ee-95ac-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.99', '--skip-admin-label'] 2024-01-13T08:15:44.316 INFO:teuthology.orchestra.run.smithi099.stderr:/bin/podman: 3.0.1 2024-01-13T08:15:44.328 INFO:teuthology.orchestra.run.smithi099.stderr:Verifying podman|docker is present... 2024-01-13T08:15:44.368 INFO:teuthology.orchestra.run.smithi099.stderr:/bin/podman: 3.0.1 2024-01-13T08:15:44.386 INFO:teuthology.orchestra.run.smithi099.stderr:Verifying lvm2 is present... 2024-01-13T08:15:44.386 INFO:teuthology.orchestra.run.smithi099.stderr:Verifying time synchronization is in place... 2024-01-13T08:15:44.393 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-01-13T08:15:44.398 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: inactive 2024-01-13T08:15:44.405 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: enabled 2024-01-13T08:15:44.410 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: active 2024-01-13T08:15:44.411 INFO:teuthology.orchestra.run.smithi099.stderr:Unit chronyd.service is enabled and running 2024-01-13T08:15:44.411 INFO:teuthology.orchestra.run.smithi099.stderr:Repeating the final host check... 2024-01-13T08:15:44.448 INFO:teuthology.orchestra.run.smithi099.stderr:/bin/podman: 3.0.1 2024-01-13T08:15:44.466 INFO:teuthology.orchestra.run.smithi099.stderr:podman (/bin/podman) version 3.0.1 is present 2024-01-13T08:15:44.466 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl is present 2024-01-13T08:15:44.466 INFO:teuthology.orchestra.run.smithi099.stderr:lvcreate is present 2024-01-13T08:15:44.472 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-01-13T08:15:44.477 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: inactive 2024-01-13T08:15:44.483 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: enabled 2024-01-13T08:15:44.488 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: active 2024-01-13T08:15:44.489 INFO:teuthology.orchestra.run.smithi099.stderr:Unit chronyd.service is enabled and running 2024-01-13T08:15:44.489 INFO:teuthology.orchestra.run.smithi099.stderr:Host looks OK 2024-01-13T08:15:44.490 INFO:teuthology.orchestra.run.smithi099.stderr:Cluster fsid: f1ba87a2-b1eb-11ee-95ac-87774f69a715 2024-01-13T08:15:44.490 INFO:teuthology.orchestra.run.smithi099.stderr:Acquiring lock 140526043811968 on /run/cephadm/f1ba87a2-b1eb-11ee-95ac-87774f69a715.lock 2024-01-13T08:15:44.490 INFO:teuthology.orchestra.run.smithi099.stderr:Lock 140526043811968 acquired on /run/cephadm/f1ba87a2-b1eb-11ee-95ac-87774f69a715.lock 2024-01-13T08:15:44.490 INFO:teuthology.orchestra.run.smithi099.stderr:Verifying IP 172.21.15.99 port 3300 ... 2024-01-13T08:15:44.490 INFO:teuthology.orchestra.run.smithi099.stderr:Verifying IP 172.21.15.99 port 6789 ... 2024-01-13T08:15:44.490 INFO:teuthology.orchestra.run.smithi099.stderr:Base mon IP(s) is [172.21.15.99:3300, 172.21.15.99:6789], mon addrv is [v2:172.21.15.99:3300,v1:172.21.15.99:6789] 2024-01-13T08:15:44.493 INFO:teuthology.orchestra.run.smithi099.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp metric 100 2024-01-13T08:15:44.493 INFO:teuthology.orchestra.run.smithi099.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.99 metric 100 2024-01-13T08:15:44.496 INFO:teuthology.orchestra.run.smithi099.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-01-13T08:15:44.496 INFO:teuthology.orchestra.run.smithi099.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-01-13T08:15:44.496 INFO:teuthology.orchestra.run.smithi099.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-01-13T08:15:44.499 INFO:teuthology.orchestra.run.smithi099.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-01-13T08:15:44.499 INFO:teuthology.orchestra.run.smithi099.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-01-13T08:15:44.499 INFO:teuthology.orchestra.run.smithi099.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-01-13T08:15:44.499 INFO:teuthology.orchestra.run.smithi099.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-01-13T08:15:44.499 INFO:teuthology.orchestra.run.smithi099.stderr:/sbin/ip: inet6 fe80::e208:a878:c5e3:8319/64 scope link noprefixroute 2024-01-13T08:15:44.499 INFO:teuthology.orchestra.run.smithi099.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-01-13T08:15:44.500 INFO:teuthology.orchestra.run.smithi099.stderr:Mon IP `172.21.15.99` is in CIDR network `172.21.0.0/20` 2024-01-13T08:15:44.500 INFO:teuthology.orchestra.run.smithi099.stderr:Mon IP `172.21.15.99` is in CIDR network `172.21.0.0/20` 2024-01-13T08:15:44.500 INFO:teuthology.orchestra.run.smithi099.stderr:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-01-13T08:15:44.504 INFO:teuthology.orchestra.run.smithi099.stderr:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-01-13T08:15:44.505 INFO:teuthology.orchestra.run.smithi099.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb... 2024-01-13T08:15:44.568 INFO:teuthology.orchestra.run.smithi099.stderr:/bin/podman: Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb... 2024-01-13T08:15:44.648 INFO:teuthology.orchestra.run.smithi099.stderr:/bin/podman: Getting image source signatures 2024-01-13T08:15:44.786 INFO:teuthology.orchestra.run.smithi099.stderr:/bin/podman: Copying blob sha256:9bc82cfe809169b39ceaa8362ba30ca8525e0ac28e35f12009a7a395fd81b5b0 2024-01-13T08:15:44.837 INFO:teuthology.orchestra.run.smithi099.stderr:/bin/podman: Copying blob sha256:6c507150f7e9f028ef978d633e702dab944bbda82bfbc46f6307495cf6dd9fc7 2024-01-13T08:16:04.599 INFO:teuthology.orchestra.run.smithi099.stderr:/bin/podman: Copying config sha256:163fd1944c31b631006e5d238574cf3311ed176b42b55b0b64e2387958260fc0 2024-01-13T08:16:04.649 INFO:teuthology.orchestra.run.smithi099.stderr:/bin/podman: Writing manifest to image destination 2024-01-13T08:16:04.649 INFO:teuthology.orchestra.run.smithi099.stderr:/bin/podman: Storing signatures 2024-01-13T08:16:14.451 INFO:teuthology.orchestra.run.smithi099.stderr:/bin/podman: 163fd1944c31b631006e5d238574cf3311ed176b42b55b0b64e2387958260fc0 2024-01-13T08:16:14.970 INFO:teuthology.orchestra.run.smithi099.stderr:ceph: ceph version 16.2.14-456-g445562ab (445562ab4bc3ddfb386936119050695810860bcb) pacific (stable) 2024-01-13T08:16:25.817 INFO:teuthology.orchestra.run.smithi099.stderr:Ceph version: ceph version 16.2.14-456-g445562ab (445562ab4bc3ddfb386936119050695810860bcb) pacific (stable) 2024-01-13T08:16:25.818 INFO:teuthology.orchestra.run.smithi099.stderr:Extracting ceph user uid/gid from container image... 2024-01-13T08:16:26.198 INFO:teuthology.orchestra.run.smithi099.stderr:stat: 167 167 2024-01-13T08:16:26.615 INFO:teuthology.orchestra.run.smithi099.stderr:Creating initial keys... 2024-01-13T08:16:26.939 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-authtool: AQDaRqJlWO3kNxAAmXJU9krJuOjbVRmDl7iuJQ== 2024-01-13T08:16:27.705 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-authtool: AQDbRqJlKQi8KRAAbFWHS3OcweVpErX2ItPHLA== 2024-01-13T08:16:28.463 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-authtool: AQDcRqJlNipbGxAAH8eYyqNSmyJTArkD/bzKrw== 2024-01-13T08:16:28.866 INFO:teuthology.orchestra.run.smithi099.stderr:Creating initial monmap... 2024-01-13T08:16:29.213 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-13T08:16:29.213 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to f1ba87a2-b1eb-11ee-95ac-87774f69a715 2024-01-13T08:16:29.213 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-13T08:16:29.658 INFO:teuthology.orchestra.run.smithi099.stderr:monmaptool for smithi099 [v2:172.21.15.99:3300,v1:172.21.15.99:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-13T08:16:29.659 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/monmaptool: set fsid to f1ba87a2-b1eb-11ee-95ac-87774f69a715 2024-01-13T08:16:29.659 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-13T08:16:29.659 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-01-13T08:16:29.659 INFO:teuthology.orchestra.run.smithi099.stderr:Creating mon... 2024-01-13T08:16:30.021 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:29.999+0000 7fcf7a03c880 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-13T08:16:30.021 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.000+0000 7fcf7a03c880 1 imported monmap: 2024-01-13T08:16:30.021 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: epoch 0 2024-01-13T08:16:30.021 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 2024-01-13T08:16:30.021 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: last_changed 2024-01-13T08:16:29.216397+0000 2024-01-13T08:16:30.021 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: created 2024-01-13T08:16:29.216397+0000 2024-01-13T08:16:30.021 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-01-13T08:16:30.021 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-01-13T08:16:30.021 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.smithi099 2024-01-13T08:16:30.022 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: 2024-01-13T08:16:30.022 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.000+0000 7fcf7a03c880 0 /usr/bin/ceph-mon: set fsid to f1ba87a2-b1eb-11ee-95ac-87774f69a715 2024-01-13T08:16:30.022 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.000+0000 7fcf7a03c880 4 rocksdb: RocksDB version: 6.8.1 2024-01-13T08:16:30.022 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: 2024-01-13T08:16:30.022 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.000+0000 7fcf7a03c880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-13T08:16:30.022 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.000+0000 7fcf7a03c880 4 rocksdb: Compile date Jan 12 2024 2024-01-13T08:16:30.022 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.000+0000 7fcf7a03c880 4 rocksdb: DB SUMMARY 2024-01-13T08:16:30.022 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: 2024-01-13T08:16:30.022 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.000+0000 7fcf7a03c880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi099/store.db dir, Total Num: 0, files: 2024-01-13T08:16:30.022 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: 2024-01-13T08:16:30.022 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.000+0000 7fcf7a03c880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi099/store.db: 2024-01-13T08:16:30.022 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: 2024-01-13T08:16:30.022 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.000+0000 7fcf7a03c880 4 rocksdb: Options.error_if_exists: 0 2024-01-13T08:16:30.022 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.000+0000 7fcf7a03c880 4 rocksdb: Options.create_if_missing: 1 2024-01-13T08:16:30.022 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.000+0000 7fcf7a03c880 4 rocksdb: Options.paranoid_checks: 1 2024-01-13T08:16:30.023 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.000+0000 7fcf7a03c880 4 rocksdb: Options.env: 0x55942b103080 2024-01-13T08:16:30.023 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.000+0000 7fcf7a03c880 4 rocksdb: Options.fs: Posix File System 2024-01-13T08:16:30.023 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.000+0000 7fcf7a03c880 4 rocksdb: Options.info_log: 0x55942c3f3380 2024-01-13T08:16:30.023 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.000+0000 7fcf7a03c880 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-13T08:16:30.023 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.000+0000 7fcf7a03c880 4 rocksdb: Options.statistics: (nil) 2024-01-13T08:16:30.023 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.000+0000 7fcf7a03c880 4 rocksdb: Options.use_fsync: 0 2024-01-13T08:16:30.023 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.000+0000 7fcf7a03c880 4 rocksdb: Options.max_log_file_size: 0 2024-01-13T08:16:30.023 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.000+0000 7fcf7a03c880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-13T08:16:30.023 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.000+0000 7fcf7a03c880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-13T08:16:30.023 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.000+0000 7fcf7a03c880 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-13T08:16:30.023 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.000+0000 7fcf7a03c880 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-13T08:16:30.023 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.000+0000 7fcf7a03c880 4 rocksdb: Options.allow_fallocate: 1 2024-01-13T08:16:30.023 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.000+0000 7fcf7a03c880 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-13T08:16:30.023 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.000+0000 7fcf7a03c880 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-13T08:16:30.023 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.000+0000 7fcf7a03c880 4 rocksdb: Options.use_direct_reads: 0 2024-01-13T08:16:30.024 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.000+0000 7fcf7a03c880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-13T08:16:30.024 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.000+0000 7fcf7a03c880 4 rocksdb: Options.create_missing_column_families: 0 2024-01-13T08:16:30.024 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.db_log_dir: 2024-01-13T08:16:30.024 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi099/store.db 2024-01-13T08:16:30.024 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-13T08:16:30.024 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.max_subcompactions: 1 2024-01-13T08:16:30.024 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.max_background_flushes: -1 2024-01-13T08:16:30.024 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-13T08:16:30.024 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-13T08:16:30.024 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-13T08:16:30.024 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-13T08:16:30.024 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-13T08:16:30.025 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.advise_random_on_open: 1 2024-01-13T08:16:30.025 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-13T08:16:30.025 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.write_buffer_manager: 0x55942c3f9980 2024-01-13T08:16:30.025 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-13T08:16:30.025 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-13T08:16:30.025 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-13T08:16:30.025 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-13T08:16:30.025 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.rate_limiter: (nil) 2024-01-13T08:16:30.025 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-13T08:16:30.025 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-13T08:16:30.025 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-13T08:16:30.025 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-13T08:16:30.025 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.unordered_write: 0 2024-01-13T08:16:30.025 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-13T08:16:30.026 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-13T08:16:30.026 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-13T08:16:30.026 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-13T08:16:30.026 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.row_cache: None 2024-01-13T08:16:30.026 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.wal_filter: None 2024-01-13T08:16:30.026 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-13T08:16:30.026 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-13T08:16:30.026 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.preserve_deletes: 0 2024-01-13T08:16:30.026 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.two_write_queues: 0 2024-01-13T08:16:30.026 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.manual_wal_flush: 0 2024-01-13T08:16:30.026 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.atomic_flush: 0 2024-01-13T08:16:30.026 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-13T08:16:30.026 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-13T08:16:30.026 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-13T08:16:30.026 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.log_readahead_size: 0 2024-01-13T08:16:30.027 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-13T08:16:30.027 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.max_background_jobs: 2 2024-01-13T08:16:30.027 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.max_background_compactions: -1 2024-01-13T08:16:30.027 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-13T08:16:30.027 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-13T08:16:30.027 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-13T08:16:30.027 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.max_total_wal_size: 0 2024-01-13T08:16:30.027 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-13T08:16:30.027 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-13T08:16:30.027 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-13T08:16:30.027 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-13T08:16:30.027 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.max_open_files: -1 2024-01-13T08:16:30.027 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.bytes_per_sync: 0 2024-01-13T08:16:30.027 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-13T08:16:30.027 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-13T08:16:30.028 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-13T08:16:30.028 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Compression algorithms supported: 2024-01-13T08:16:30.028 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-13T08:16:30.028 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: kZSTD supported: 0 2024-01-13T08:16:30.028 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: kXpressCompression supported: 0 2024-01-13T08:16:30.028 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-13T08:16:30.028 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: kLZ4Compression supported: 1 2024-01-13T08:16:30.028 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: kBZip2Compression supported: 0 2024-01-13T08:16:30.028 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: kZlibCompression supported: 1 2024-01-13T08:16:30.028 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: kSnappyCompression supported: 1 2024-01-13T08:16:30.028 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-13T08:16:30.028 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-01-13T08:16:30.028 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: 2024-01-13T08:16:30.028 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi099/store.db/MANIFEST-000001 2024-01-13T08:16:30.028 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: 2024-01-13T08:16:30.029 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-13T08:16:30.029 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: 2024-01-13T08:16:30.029 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-13T08:16:30.029 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.merge_operator: 2024-01-13T08:16:30.029 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.compaction_filter: None 2024-01-13T08:16:30.029 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.compaction_filter_factory: None 2024-01-13T08:16:30.029 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-13T08:16:30.029 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-13T08:16:30.029 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55942c303d30) 2024-01-13T08:16:30.029 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-01-13T08:16:30.029 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-13T08:16:30.029 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-13T08:16:30.029 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-01-13T08:16:30.029 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: index_type: 0 2024-01-13T08:16:30.029 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-01-13T08:16:30.030 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-01-13T08:16:30.030 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-01-13T08:16:30.030 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-01-13T08:16:30.030 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: checksum: 1 2024-01-13T08:16:30.030 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-01-13T08:16:30.030 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: block_cache: 0x55942c33cd10 2024-01-13T08:16:30.030 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-01-13T08:16:30.030 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-01-13T08:16:30.030 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-01-13T08:16:30.030 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-01-13T08:16:30.030 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-01-13T08:16:30.030 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-01-13T08:16:30.030 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-01-13T08:16:30.030 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-01-13T08:16:30.030 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-01-13T08:16:30.030 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-01-13T08:16:30.031 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-01-13T08:16:30.031 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-01-13T08:16:30.031 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-01-13T08:16:30.031 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-01-13T08:16:30.031 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-01-13T08:16:30.031 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-13T08:16:30.031 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-01-13T08:16:30.031 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-01-13T08:16:30.031 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-01-13T08:16:30.031 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: format_version: 2 2024-01-13T08:16:30.031 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-01-13T08:16:30.031 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: block_align: 0 2024-01-13T08:16:30.031 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: 2024-01-13T08:16:30.031 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-13T08:16:30.031 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-13T08:16:30.032 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.compression: NoCompression 2024-01-13T08:16:30.032 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-13T08:16:30.032 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-13T08:16:30.032 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-13T08:16:30.032 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.num_levels: 7 2024-01-13T08:16:30.032 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-13T08:16:30.032 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-13T08:16:30.032 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-13T08:16:30.032 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-13T08:16:30.032 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-13T08:16:30.032 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-13T08:16:30.032 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-13T08:16:30.032 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-13T08:16:30.032 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-13T08:16:30.033 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-13T08:16:30.033 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.compression_opts.level: 32767 2024-01-13T08:16:30.033 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-13T08:16:30.033 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-13T08:16:30.033 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-13T08:16:30.033 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.compression_opts.enabled: false 2024-01-13T08:16:30.033 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-13T08:16:30.033 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-13T08:16:30.033 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-13T08:16:30.033 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-13T08:16:30.033 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-13T08:16:30.033 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-13T08:16:30.033 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-13T08:16:30.033 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-13T08:16:30.033 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-13T08:16:30.033 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-13T08:16:30.034 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-13T08:16:30.034 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-13T08:16:30.034 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-13T08:16:30.034 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-13T08:16:30.034 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-13T08:16:30.034 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-13T08:16:30.034 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-13T08:16:30.034 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.arena_block_size: 4194304 2024-01-13T08:16:30.034 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-13T08:16:30.034 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-13T08:16:30.034 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-13T08:16:30.034 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-13T08:16:30.034 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-13T08:16:30.034 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-13T08:16:30.034 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-13T08:16:30.035 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-13T08:16:30.035 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-13T08:16:30.035 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-13T08:16:30.035 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-13T08:16:30.035 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-13T08:16:30.035 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-13T08:16:30.035 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-13T08:16:30.035 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.table_properties_collectors: 2024-01-13T08:16:30.035 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.inplace_update_support: 0 2024-01-13T08:16:30.035 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-13T08:16:30.035 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-13T08:16:30.035 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-13T08:16:30.035 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-13T08:16:30.035 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.bloom_locality: 0 2024-01-13T08:16:30.035 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.max_successive_merges: 0 2024-01-13T08:16:30.035 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-13T08:16:30.036 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-13T08:16:30.036 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.force_consistency_checks: 0 2024-01-13T08:16:30.036 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-13T08:16:30.036 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.ttl: 2592000 2024-01-13T08:16:30.036 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.001+0000 7fcf7a03c880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-13T08:16:30.036 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.002+0000 7fcf7a03c880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi099/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-13T08:16:30.036 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: 2024-01-13T08:16:30.036 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.002+0000 7fcf7a03c880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-01-13T08:16:30.036 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: 2024-01-13T08:16:30.036 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.003+0000 7fcf7a03c880 4 rocksdb: DB pointer 0x55942c407800 2024-01-13T08:16:30.036 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.003+0000 7fcf62eb1700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-13T08:16:30.036 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.003+0000 7fcf62eb1700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-13T08:16:30.036 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-01-13T08:16:30.036 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T08:16:30.037 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-13T08:16:30.037 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-13T08:16:30.037 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-13T08:16:30.037 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-13T08:16:30.037 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-13T08:16:30.037 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-13T08:16:30.037 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: 2024-01-13T08:16:30.037 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-13T08:16:30.037 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T08:16:30.037 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T08:16:30.037 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-13T08:16:30.037 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-13T08:16:30.037 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: 2024-01-13T08:16:30.037 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-13T08:16:30.037 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T08:16:30.038 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T08:16:30.038 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T08:16:30.038 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-01-13T08:16:30.038 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-13T08:16:30.038 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-01-13T08:16:30.038 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-01-13T08:16:30.038 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-01-13T08:16:30.038 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T08:16:30.038 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T08:16:30.038 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-13T08:16:30.038 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: 2024-01-13T08:16:30.038 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-01-13T08:16:30.038 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: 2024-01-13T08:16:30.038 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-13T08:16:30.038 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T08:16:30.039 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T08:16:30.039 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-13T08:16:30.039 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-13T08:16:30.039 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: 2024-01-13T08:16:30.039 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-13T08:16:30.039 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T08:16:30.039 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T08:16:30.039 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T08:16:30.039 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-01-13T08:16:30.039 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-13T08:16:30.039 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-01-13T08:16:30.039 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-01-13T08:16:30.039 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-01-13T08:16:30.039 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T08:16:30.039 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T08:16:30.040 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-13T08:16:30.040 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: 2024-01-13T08:16:30.040 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-01-13T08:16:30.040 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: 2024-01-13T08:16:30.040 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.004+0000 7fcf7a03c880 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-01-13T08:16:30.040 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.004+0000 7fcf7a03c880 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-01-13T08:16:30.040 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph-mon: debug 2024-01-13T08:16:30.004+0000 7fcf7a03c880 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi099 for mon.smithi099 2024-01-13T08:16:30.469 INFO:teuthology.orchestra.run.smithi099.stderr:create mon.smithi099 on 2024-01-13T08:16:30.573 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-01-13T08:16:30.669 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-f1ba87a2-b1eb-11ee-95ac-87774f69a715.target → /etc/systemd/system/ceph-f1ba87a2-b1eb-11ee-95ac-87774f69a715.target. 2024-01-13T08:16:30.670 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-f1ba87a2-b1eb-11ee-95ac-87774f69a715.target → /etc/systemd/system/ceph-f1ba87a2-b1eb-11ee-95ac-87774f69a715.target. 2024-01-13T08:16:30.875 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: Failed to reset failed state of unit ceph-f1ba87a2-b1eb-11ee-95ac-87774f69a715@mon.smithi099.service: Unit ceph-f1ba87a2-b1eb-11ee-95ac-87774f69a715@mon.smithi099.service not loaded. 2024-01-13T08:16:30.881 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: Created symlink /etc/systemd/system/ceph-f1ba87a2-b1eb-11ee-95ac-87774f69a715.target.wants/ceph-f1ba87a2-b1eb-11ee-95ac-87774f69a715@mon.smithi099.service → /etc/systemd/system/ceph-f1ba87a2-b1eb-11ee-95ac-87774f69a715@.service. 2024-01-13T08:16:31.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:30 smithi099 systemd[1]: Starting Ceph mon.smithi099 for f1ba87a2-b1eb-11ee-95ac-87774f69a715... 2024-01-13T08:16:31.510 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: disabled 2024-01-13T08:16:31.518 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: inactive 2024-01-13T08:16:31.518 INFO:teuthology.orchestra.run.smithi099.stderr:firewalld.service is not enabled 2024-01-13T08:16:31.518 INFO:teuthology.orchestra.run.smithi099.stderr:Not possible to enable service . firewalld.service is not available 2024-01-13T08:16:31.519 INFO:teuthology.orchestra.run.smithi099.stderr:Waiting for mon to start... 2024-01-13T08:16:31.519 INFO:teuthology.orchestra.run.smithi099.stderr:Waiting for mon... 2024-01-13T08:16:31.690 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:31 smithi099 bash[32288]: edc1fdfc435346c17e2f03fc5c313a02f922e0ac9fb196dd7f3567011e9c5869 2024-01-13T08:16:31.690 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:31 smithi099 systemd[1]: Started Ceph mon.smithi099 for f1ba87a2-b1eb-11ee-95ac-87774f69a715. 2024-01-13T08:16:31.690 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:31 smithi099 conmon[32328]: cluster 2024-01-13T08:16:31.542909+0000 mon.smithi099 (mon.0) 0 : 2024-01-13T08:16:31.690 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:31 smithi099 conmon[32328]: [INF] mkfs f1ba87a2-b1eb-11ee-95ac-87774f69a715 2024-01-13T08:16:31.690 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:31 smithi099 conmon[32328]: cluster 2024-01-13T08:16:31.528859+0000 mon.smithi099 (mon.0) 1 : cluster [INF] mon.smithi099 is new leader, mons smithi099 in quorum (ranks 0) 2024-01-13T08:16:31.977 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: cluster: 2024-01-13T08:16:31.978 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: id: f1ba87a2-b1eb-11ee-95ac-87774f69a715 2024-01-13T08:16:31.978 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-01-13T08:16:31.978 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: 2024-01-13T08:16:31.978 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: services: 2024-01-13T08:16:31.978 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi099 (age 0.439141s) 2024-01-13T08:16:31.978 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: mgr: no daemons active 2024-01-13T08:16:31.978 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-01-13T08:16:31.978 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: 2024-01-13T08:16:31.978 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: data: 2024-01-13T08:16:31.978 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-01-13T08:16:31.978 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-01-13T08:16:31.978 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-01-13T08:16:31.978 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: pgs: 2024-01-13T08:16:31.978 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: 2024-01-13T08:16:32.589 INFO:teuthology.orchestra.run.smithi099.stderr:mon is available 2024-01-13T08:16:32.589 INFO:teuthology.orchestra.run.smithi099.stderr:Assimilating anything we can from ceph.conf... 2024-01-13T08:16:32.804 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:32 smithi099 conmon[32328]: cluster 2024-01-13T08:16:31.539577+0000 mon.smithi099 (mon.0) 2 : cluster [INF] mon.smithi099 is new leader, mons smithi099 in quorum (ranks 0) 2024-01-13T08:16:32.805 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:32 smithi099 conmon[32328]: cluster 2024-01-13T08:16:31.539790+0000 mon.smithi099 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi099=[v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0]} removed_ranks: {} 2024-01-13T08:16:32.805 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:32 smithi099 conmon[32328]: cluster 2024-01-13T08:16:31.548816+0000 mon.smithi099 (mon.0) 4 : cluster [DBG] fsmap 2024-01-13T08:16:32.805 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:32 smithi099 conmon[32328]: cluster 2024-01-13T08:16:31.555776+0000 mon.smithi099 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-01-13T08:16:32.805 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:32 smithi099 conmon[32328]: cluster 2024-01-13T08:16:31.556389+0000 mon.smithi099 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-01-13T08:16:32.805 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:32 smithi099 conmon[32328]: audit 2024-01-13T08:16:31.978666+0000 mon.smithi099 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.99:0/2736171115' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-01-13T08:16:33.052 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: 2024-01-13T08:16:33.052 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: [global] 2024-01-13T08:16:33.052 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: fsid = f1ba87a2-b1eb-11ee-95ac-87774f69a715 2024-01-13T08:16:33.052 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.99:3300,v1:172.21.15.99:6789] 2024-01-13T08:16:33.052 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-01-13T08:16:33.052 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-01-13T08:16:33.052 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-01-13T08:16:33.052 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-01-13T08:16:33.052 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: 2024-01-13T08:16:33.052 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: [mgr] 2024-01-13T08:16:33.052 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-01-13T08:16:33.052 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: 2024-01-13T08:16:33.053 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: [osd] 2024-01-13T08:16:33.053 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-01-13T08:16:33.053 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-01-13T08:16:33.522 INFO:teuthology.orchestra.run.smithi099.stderr:Generating new minimal ceph.conf... 2024-01-13T08:16:33.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:33 smithi099 conmon[32328]: audit 2024-01-13T08:16:33.050853+0000 mon.smithi099 (mon.0) 8 2024-01-13T08:16:33.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:33 smithi099 conmon[32328]: : audit [INF] from='client.? 172.21.15.99:0/1223122043' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-13T08:16:33.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:33 smithi099 conmon[32328]: audit 2024-01-13T08:16:33.053551+0000 mon.smithi099 (mon.0) 9 : audit [INF] from='client.? 172.21.15.99:0/1223122043' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-01-13T08:16:34.556 INFO:teuthology.orchestra.run.smithi099.stderr:Restarting the monitor... 2024-01-13T08:16:34.827 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:34 smithi099 conmon[32328]: audit 2024-01-13T08:16:34.035486+0000 mon.smithi099 (mon 2024-01-13T08:16:34.827 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:34 smithi099 conmon[32328]: .0) 10 : audit [DBG] from='client.? 172.21.15.99:0/3201484914' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:16:34.827 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:34 smithi099 systemd[1]: Stopping Ceph mon.smithi099 for f1ba87a2-b1eb-11ee-95ac-87774f69a715... 2024-01-13T08:16:34.827 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:34 smithi099 bash[32831]: Error: no container with name or ID ceph-f1ba87a2-b1eb-11ee-95ac-87774f69a715-mon.smithi099 found: no such container 2024-01-13T08:16:34.827 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:34 smithi099 conmon[32328]: debug 2024-01-13T08:16:34.829+0000 7f7aaed0a700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.smithi099 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-13T08:16:34.827 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:34 smithi099 conmon[32328]: debug 2024-01-13T08:16:34.829+0000 7f7aaed0a700 -1 mon.smithi099@0(leader) e1 *** Got Signal Terminated *** 2024-01-13T08:16:35.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 bash[32948]: edc1fdfc435346c17e2f03fc5c313a02f922e0ac9fb196dd7f3567011e9c5869 2024-01-13T08:16:35.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 systemd[1]: ceph-f1ba87a2-b1eb-11ee-95ac-87774f69a715@mon.smithi099.service: Succeeded. 2024-01-13T08:16:35.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 systemd[1]: Stopped Ceph mon.smithi099 for f1ba87a2-b1eb-11ee-95ac-87774f69a715. 2024-01-13T08:16:35.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 systemd[1]: Starting Ceph mon.smithi099 for f1ba87a2-b1eb-11ee-95ac-87774f69a715... 2024-01-13T08:16:35.925 INFO:teuthology.orchestra.run.smithi099.stderr:Setting public_network to 172.21.0.0/20 in mon config section 2024-01-13T08:16:36.176 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.916+0000 7f6905f48880 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-13T08:16:36.176 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.916+0000 7f6905f48880 0 ceph version 16.2.14-456-g445562ab (445562ab4bc3ddfb386936119050695810860bcb) pacific (stable), process ceph-mon, pid 7 2024-01-13T08:16:36.177 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.916+0000 7f6905f48880 0 pidfile_write: ignore empty --pid-file 2024-01-13T08:16:36.177 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 0 load: jerasure load: lrc load: isa 2024-01-13T08:16:36.177 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: RocksDB version: 6.8.1 2024-01-13T08:16:36.177 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: 2024-01-13T08:16:36.177 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-13T08:16:36.178 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Compile date Jan 12 2024 2024-01-13T08:16:36.178 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: DB SUMMARY 2024-01-13T08:16:36.178 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: 2024-01-13T08:16:36.178 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: CURRENT file: CURRENT 2024-01-13T08:16:36.178 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: 2024-01-13T08:16:36.178 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: IDENTITY file: IDENTITY 2024-01-13T08:16:36.178 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: 2024-01-13T08:16:36.178 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-01-13T08:16:36.178 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: 2024-01-13T08:16:36.178 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi099/store.db dir, Total Num: 1, files: 000004.sst 2024-01-13T08:16:36.178 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: 2024-01-13T08:16:36.178 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi099/store.db: 000006.log size: 86507 ; 2024-01-13T08:16:36.178 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: 2024-01-13T08:16:36.178 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.error_if_exists: 0 2024-01-13T08:16:36.178 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 bash[33272]: 7133571b7b4ab227998a844cd346f819d7957e42c1704fcd088eb721b2e8608d 2024-01-13T08:16:36.179 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.create_if_missing: 0 2024-01-13T08:16:36.179 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.paranoid_checks: 1 2024-01-13T08:16:36.179 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.env: 0x556b1624e080 2024-01-13T08:16:36.179 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.fs: Posix File System 2024-01-13T08:16:36.179 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.info_log: 0x556b18161d20 2024-01-13T08:16:36.179 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-13T08:16:36.179 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.statistics: (nil) 2024-01-13T08:16:36.179 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.use_fsync: 0 2024-01-13T08:16:36.179 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.max_log_file_size: 0 2024-01-13T08:16:36.179 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-13T08:16:36.179 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-13T08:16:36.179 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-13T08:16:36.179 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-13T08:16:36.179 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.allow_fallocate: 1 2024-01-13T08:16:36.179 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-13T08:16:36.180 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-13T08:16:36.180 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.use_direct_reads: 0 2024-01-13T08:16:36.180 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-13T08:16:36.180 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.create_missing_column_families: 0 2024-01-13T08:16:36.180 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.db_log_dir: 2024-01-13T08:16:36.180 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi099/store.db 2024-01-13T08:16:36.180 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-13T08:16:36.180 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.max_subcompactions: 1 2024-01-13T08:16:36.180 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.max_background_flushes: -1 2024-01-13T08:16:36.180 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-13T08:16:36.180 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-13T08:16:36.180 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-13T08:16:36.180 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-13T08:16:36.180 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-13T08:16:36.181 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.advise_random_on_open: 1 2024-01-13T08:16:36.181 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-13T08:16:36.181 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.write_buffer_manager: 0x556b18db8480 2024-01-13T08:16:36.181 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-13T08:16:36.181 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-13T08:16:36.181 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-13T08:16:36.181 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-13T08:16:36.181 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.rate_limiter: (nil) 2024-01-13T08:16:36.181 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-13T08:16:36.181 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-13T08:16:36.181 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-13T08:16:36.181 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-13T08:16:36.181 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.unordered_write: 0 2024-01-13T08:16:36.181 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-13T08:16:36.182 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-13T08:16:36.182 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-13T08:16:36.182 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-13T08:16:36.182 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.row_cache: None 2024-01-13T08:16:36.182 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.wal_filter: None 2024-01-13T08:16:36.182 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-13T08:16:36.182 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-13T08:16:36.182 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.preserve_deletes: 0 2024-01-13T08:16:36.182 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.two_write_queues: 0 2024-01-13T08:16:36.182 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.manual_wal_flush: 0 2024-01-13T08:16:36.182 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.atomic_flush: 0 2024-01-13T08:16:36.182 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-13T08:16:36.182 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-13T08:16:36.182 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-13T08:16:36.182 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.log_readahead_size: 0 2024-01-13T08:16:36.183 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-13T08:16:36.183 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.max_background_jobs: 2 2024-01-13T08:16:36.183 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.max_background_compactions: -1 2024-01-13T08:16:36.183 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-13T08:16:36.183 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-13T08:16:36.183 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-13T08:16:36.183 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.max_total_wal_size: 0 2024-01-13T08:16:36.183 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-13T08:16:36.183 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-13T08:16:36.183 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-13T08:16:36.183 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-13T08:16:36.183 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.max_open_files: -1 2024-01-13T08:16:36.183 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.bytes_per_sync: 0 2024-01-13T08:16:36.183 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-13T08:16:36.183 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-13T08:16:36.184 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-13T08:16:36.184 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Compression algorithms supported: 2024-01-13T08:16:36.184 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-13T08:16:36.184 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: kZSTD supported: 0 2024-01-13T08:16:36.184 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: kXpressCompression supported: 0 2024-01-13T08:16:36.184 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-13T08:16:36.184 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: kLZ4Compression supported: 1 2024-01-13T08:16:36.184 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: kBZip2Compression supported: 0 2024-01-13T08:16:36.184 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: kZlibCompression supported: 1 2024-01-13T08:16:36.184 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: kSnappyCompression supported: 1 2024-01-13T08:16:36.184 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-13T08:16:36.184 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi099/store.db/MANIFEST-000005 2024-01-13T08:16:36.184 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: 2024-01-13T08:16:36.184 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-13T08:16:36.184 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: 2024-01-13T08:16:36.185 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-13T08:16:36.185 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.merge_operator: 2024-01-13T08:16:36.185 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.compaction_filter: None 2024-01-13T08:16:36.185 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.compaction_filter_factory: None 2024-01-13T08:16:36.185 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-13T08:16:36.185 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-13T08:16:36.185 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556b1810bd28) 2024-01-13T08:16:36.185 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: cache_index_and_filter_blocks: 1 2024-01-13T08:16:36.185 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-13T08:16:36.185 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-13T08:16:36.185 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: pin_top_level_index_and_filter: 1 2024-01-13T08:16:36.185 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: index_type: 0 2024-01-13T08:16:36.185 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: data_block_index_type: 0 2024-01-13T08:16:36.186 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: index_shortening: 1 2024-01-13T08:16:36.186 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: data_block_hash_table_util_ratio: 0.750000 2024-01-13T08:16:36.186 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: hash_index_allow_collision: 1 2024-01-13T08:16:36.186 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: checksum: 1 2024-01-13T08:16:36.186 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: no_block_cache: 0 2024-01-13T08:16:36.186 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: block_cache: 0x556b18144f10 2024-01-13T08:16:36.186 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: block_cache_name: BinnedLRUCache 2024-01-13T08:16:36.186 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: block_cache_options: 2024-01-13T08:16:36.186 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: capacity : 536870912 2024-01-13T08:16:36.186 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: num_shard_bits : 4 2024-01-13T08:16:36.186 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: strict_capacity_limit : 0 2024-01-13T08:16:36.186 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: high_pri_pool_ratio: 0.000 2024-01-13T08:16:36.186 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: block_cache_compressed: (nil) 2024-01-13T08:16:36.186 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: persistent_cache: (nil) 2024-01-13T08:16:36.186 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: block_size: 4096 2024-01-13T08:16:36.187 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: block_size_deviation: 10 2024-01-13T08:16:36.187 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: block_restart_interval: 16 2024-01-13T08:16:36.187 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: index_block_restart_interval: 1 2024-01-13T08:16:36.187 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: metadata_block_size: 4096 2024-01-13T08:16:36.187 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: partition_filters: 0 2024-01-13T08:16:36.187 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: use_delta_encoding: 1 2024-01-13T08:16:36.187 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-13T08:16:36.187 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: whole_key_filtering: 1 2024-01-13T08:16:36.187 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: verify_compression: 0 2024-01-13T08:16:36.187 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: read_amp_bytes_per_bit: 0 2024-01-13T08:16:36.187 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: format_version: 2 2024-01-13T08:16:36.187 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: enable_index_compression: 1 2024-01-13T08:16:36.187 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: block_align: 0 2024-01-13T08:16:36.187 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: 2024-01-13T08:16:36.188 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-13T08:16:36.188 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.921+0000 7f6905f48880 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-13T08:16:36.188 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.compression: NoCompression 2024-01-13T08:16:36.188 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-13T08:16:36.188 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-13T08:16:36.188 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-13T08:16:36.188 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.num_levels: 7 2024-01-13T08:16:36.188 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-13T08:16:36.188 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-13T08:16:36.188 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-13T08:16:36.188 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-13T08:16:36.188 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-13T08:16:36.188 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-13T08:16:36.188 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-13T08:16:36.189 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-13T08:16:36.189 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-13T08:16:36.189 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-13T08:16:36.189 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.compression_opts.level: 32767 2024-01-13T08:16:36.189 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-13T08:16:36.189 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-13T08:16:36.189 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-13T08:16:36.189 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.compression_opts.enabled: false 2024-01-13T08:16:36.189 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-13T08:16:36.189 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-13T08:16:36.189 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-13T08:16:36.189 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-13T08:16:36.189 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-13T08:16:36.189 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-13T08:16:36.189 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-13T08:16:36.190 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-13T08:16:36.190 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-13T08:16:36.190 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-13T08:16:36.190 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-13T08:16:36.190 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-13T08:16:36.190 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-13T08:16:36.190 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-13T08:16:36.190 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-13T08:16:36.190 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-13T08:16:36.190 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-13T08:16:36.190 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.arena_block_size: 4194304 2024-01-13T08:16:36.190 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-13T08:16:36.190 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-13T08:16:36.190 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-13T08:16:36.190 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-13T08:16:36.191 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-13T08:16:36.191 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-13T08:16:36.191 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-13T08:16:36.191 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-13T08:16:36.191 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-13T08:16:36.191 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-13T08:16:36.191 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-13T08:16:36.191 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-13T08:16:36.191 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-13T08:16:36.191 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-13T08:16:36.191 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.table_properties_collectors: 2024-01-13T08:16:36.191 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.inplace_update_support: 0 2024-01-13T08:16:36.191 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-13T08:16:36.191 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-13T08:16:36.191 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-13T08:16:36.192 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-13T08:16:36.192 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.bloom_locality: 0 2024-01-13T08:16:36.192 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.max_successive_merges: 0 2024-01-13T08:16:36.192 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-13T08:16:36.192 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-13T08:16:36.192 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.force_consistency_checks: 0 2024-01-13T08:16:36.192 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-13T08:16:36.192 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.ttl: 2592000 2024-01-13T08:16:36.192 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-13T08:16:36.192 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi099/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-13T08:16:36.192 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: 2024-01-13T08:16:36.192 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-01-13T08:16:36.192 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: 2024-01-13T08:16:36.192 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705133795923864, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-01-13T08:16:36.192 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.922+0000 7f6905f48880 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-01-13T08:16:36.193 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.923+0000 7f6905f48880 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-01-13T08:16:36.193 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.923+0000 7f6905f48880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705133795924666, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 83380, "table_properties": {"data_size": 81647, "index_size": 338, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 581, "raw_key_size": 9331, "raw_average_key_size": 47, "raw_value_size": 76159, "raw_average_value_size": 384, "num_data_blocks": 11, "num_entries": 198, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1705133795, "oldest_key_time": 3, "file_creation_time": 0}} 2024-01-13T08:16:36.193 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.923+0000 7f6905f48880 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-01-13T08:16:36.193 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: 2024-01-13T08:16:36.193 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.923+0000 7f6905f48880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705133795925074, "job": 1, "event": "recovery_finished"} 2024-01-13T08:16:36.193 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.924+0000 7f6905f48880 4 rocksdb: DB pointer 0x556b1820f800 2024-01-13T08:16:36.193 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.924+0000 7f68ecd3c700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-13T08:16:36.193 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.924+0000 7f68ecd3c700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-13T08:16:36.193 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: ** DB Stats ** 2024-01-13T08:16:36.193 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T08:16:36.193 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-13T08:16:36.193 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-13T08:16:36.193 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-13T08:16:36.193 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-13T08:16:36.193 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-13T08:16:36.194 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-13T08:16:36.194 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: 2024-01-13T08:16:36.194 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: ** Compaction Stats [default] ** 2024-01-13T08:16:36.194 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T08:16:36.194 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T08:16:36.194 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: L0 2/0 83.12 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 141.0 0.00 0.00 1 0.001 0 0 2024-01-13T08:16:36.194 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: Sum 2/0 83.12 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 141.0 0.00 0.00 1 0.001 0 0 2024-01-13T08:16:36.194 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 141.0 0.00 0.00 1 0.001 0 0 2024-01-13T08:16:36.194 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: 2024-01-13T08:16:36.194 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: ** Compaction Stats [default] ** 2024-01-13T08:16:36.194 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T08:16:36.194 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T08:16:36.194 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 141.0 0.00 0.00 1 0.001 0 0 2024-01-13T08:16:36.194 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T08:16:36.194 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-13T08:16:36.195 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-13T08:16:36.195 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: AddFile(Total Files): cumulative 0, interval 0 2024-01-13T08:16:36.195 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-13T08:16:36.195 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: AddFile(Keys): cumulative 0, interval 0 2024-01-13T08:16:36.195 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: Cumulative compaction: 0.00 GB write, 29.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T08:16:36.195 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: Interval compaction: 0.00 GB write, 29.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T08:16:36.195 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-13T08:16:36.195 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: 2024-01-13T08:16:36.195 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: ** File Read Latency Histogram By Level [default] ** 2024-01-13T08:16:36.195 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: 2024-01-13T08:16:36.195 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: ** Compaction Stats [default] ** 2024-01-13T08:16:36.195 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T08:16:36.195 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T08:16:36.195 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: L0 2/0 83.12 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 141.0 0.00 0.00 1 0.001 0 0 2024-01-13T08:16:36.195 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: Sum 2/0 83.12 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 141.0 0.00 0.00 1 0.001 0 0 2024-01-13T08:16:36.196 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-13T08:16:36.196 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: 2024-01-13T08:16:36.196 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: ** Compaction Stats [default] ** 2024-01-13T08:16:36.196 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-13T08:16:36.196 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-13T08:16:36.196 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 141.0 0.00 0.00 1 0.001 0 0 2024-01-13T08:16:36.196 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-13T08:16:36.196 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-13T08:16:36.196 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-13T08:16:36.196 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: AddFile(Total Files): cumulative 0, interval 0 2024-01-13T08:16:36.196 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-13T08:16:36.196 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: AddFile(Keys): cumulative 0, interval 0 2024-01-13T08:16:36.196 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: Cumulative compaction: 0.00 GB write, 28.72 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T08:16:36.196 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-13T08:16:36.197 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-13T08:16:36.197 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: 2024-01-13T08:16:36.198 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: ** File Read Latency Histogram By Level [default] ** 2024-01-13T08:16:36.199 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: 2024-01-13T08:16:36.199 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.925+0000 7f6905f48880 0 starting mon.smithi099 rank 0 at public addrs [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] at bind addrs [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi099 fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 2024-01-13T08:16:36.199 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 systemd[1]: Started Ceph mon.smithi099 for f1ba87a2-b1eb-11ee-95ac-87774f69a715. 2024-01-13T08:16:36.199 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.926+0000 7f6905f48880 1 mon.smithi099@-1(???) e1 preinit fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 2024-01-13T08:16:36.199 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.926+0000 7f6905f48880 0 mon.smithi099@-1(???).mds e1 new map 2024-01-13T08:16:36.199 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.926+0000 7f6905f48880 0 mon.smithi099@-1(???).mds e1 print_map 2024-01-13T08:16:36.199 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: e1 2024-01-13T08:16:36.199 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: enable_multiple, ever_enabled_multiple: 1,1 2024-01-13T08:16:36.199 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-13T08:16:36.199 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: legacy client fscid: -1 2024-01-13T08:16:36.199 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: 2024-01-13T08:16:36.199 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: No filesystems configured 2024-01-13T08:16:36.199 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: 2024-01-13T08:16:36.199 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.926+0000 7f6905f48880 0 mon.smithi099@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-01-13T08:16:36.199 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.926+0000 7f6905f48880 0 mon.smithi099@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-13T08:16:36.200 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.926+0000 7f6905f48880 0 mon.smithi099@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-13T08:16:36.200 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.926+0000 7f6905f48880 0 mon.smithi099@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-13T08:16:36.200 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: audit 2024-01-13T08:16:36.200 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: 2024-01-13T08:16:34.035486+0000 mon.smithi099 (mon.0) 10 : audit [DBG] 2024-01-13T08:16:36.200 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: from='client.? 172.21.15.99:0/3201484914' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:16:36.200 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:35 smithi099 conmon[33313]: debug 2024-01-13T08:16:35.926+0000 7f6905f48880 1 mon.smithi099@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-01-13T08:16:36.200 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:36 smithi099 conmon[33313]: cluster 2024-01-13T08:16:35.945966+0000 mon.smithi099 (mon.0) 1 : cluster [INF] mon.smithi099 is new leader, mons smithi099 in quorum (ranks 0) 2024-01-13T08:16:36.200 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:36 smithi099 conmon[33313]: cluster 2024-01-13T08 2024-01-13T08:16:36.200 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:36 smithi099 conmon[33313]: :16:35.946045+0000 mon.smithi099 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi099=[v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0]} removed_ranks: {} 2024-01-13T08:16:36.200 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:36 smithi099 conmon[33313]: cluster 2024-01-13T08:16:35.957617+0000 mon.smithi099 (mon.0) 3 : cluster [DBG] fsmap 2024-01-13T08:16:36.200 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:36 smithi099 conmon[33313]: cluster 2024-01-13T08:16:35.957660+0000 mon.smithi099 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-01-13T08:16:36.200 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:36 smithi099 conmon[33313]: cluster 2024-01-13T08:16:35.958192+0000 mon.smithi099 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-01-13T08:16:37.015 INFO:teuthology.orchestra.run.smithi099.stderr:Wrote config to /etc/ceph/ceph.conf 2024-01-13T08:16:37.016 INFO:teuthology.orchestra.run.smithi099.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-01-13T08:16:37.016 INFO:teuthology.orchestra.run.smithi099.stderr:Creating mgr... 2024-01-13T08:16:37.016 INFO:teuthology.orchestra.run.smithi099.stderr:Verifying port 9283 ... 2024-01-13T08:16:37.143 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: Failed to reset failed state of unit ceph-f1ba87a2-b1eb-11ee-95ac-87774f69a715@mgr.smithi099.oodult.service: Unit ceph-f1ba87a2-b1eb-11ee-95ac-87774f69a715@mgr.smithi099.oodult.service not loaded. 2024-01-13T08:16:37.149 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: Created symlink /etc/systemd/system/ceph-f1ba87a2-b1eb-11ee-95ac-87774f69a715.target.wants/ceph-f1ba87a2-b1eb-11ee-95ac-87774f69a715@mgr.smithi099.oodult.service → /etc/systemd/system/ceph-f1ba87a2-b1eb-11ee-95ac-87774f69a715@.service. 2024-01-13T08:16:37.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:37 smithi099 conmon[33313]: audit 2024-01-13T08:16:36.444140 2024-01-13T08:16:37.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:37 smithi099 conmon[33313]: +0000 mon.smithi099 (mon.0) 6 : audit [INF] from='client.? 172.21.15.99:0/265485950' entity='client.admin' 2024-01-13T08:16:37.776 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: disabled 2024-01-13T08:16:37.783 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: inactive 2024-01-13T08:16:37.784 INFO:teuthology.orchestra.run.smithi099.stderr:firewalld.service is not enabled 2024-01-13T08:16:37.784 INFO:teuthology.orchestra.run.smithi099.stderr:Not possible to enable service . firewalld.service is not available 2024-01-13T08:16:37.791 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: disabled 2024-01-13T08:16:37.799 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: inactive 2024-01-13T08:16:37.800 INFO:teuthology.orchestra.run.smithi099.stderr:firewalld.service is not enabled 2024-01-13T08:16:37.800 INFO:teuthology.orchestra.run.smithi099.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-01-13T08:16:37.800 INFO:teuthology.orchestra.run.smithi099.stderr:Waiting for mgr to start... 2024-01-13T08:16:37.800 INFO:teuthology.orchestra.run.smithi099.stderr:Waiting for mgr... 2024-01-13T08:16:38.300 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: 2024-01-13T08:16:38.300 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: { 2024-01-13T08:16:38.300 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "fsid": "f1ba87a2-b1eb-11ee-95ac-87774f69a715", 2024-01-13T08:16:38.301 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "health": { 2024-01-13T08:16:38.301 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-13T08:16:38.301 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "checks": {}, 2024-01-13T08:16:38.301 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "mutes": [] 2024-01-13T08:16:38.301 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-13T08:16:38.301 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-13T08:16:38.301 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "quorum": [ 2024-01-13T08:16:38.301 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: 0 2024-01-13T08:16:38.301 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: ], 2024-01-13T08:16:38.301 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-13T08:16:38.301 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "smithi099" 2024-01-13T08:16:38.301 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: ], 2024-01-13T08:16:38.301 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-01-13T08:16:38.302 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "monmap": { 2024-01-13T08:16:38.302 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T08:16:38.302 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-13T08:16:38.302 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-13T08:16:38.302 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-13T08:16:38.302 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "osdmap": { 2024-01-13T08:16:38.302 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T08:16:38.302 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-13T08:16:38.302 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-13T08:16:38.302 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-13T08:16:38.302 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-13T08:16:38.307 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-13T08:16:38.307 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-13T08:16:38.307 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-13T08:16:38.307 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "pgmap": { 2024-01-13T08:16:38.307 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-13T08:16:38.307 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-13T08:16:38.307 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-13T08:16:38.307 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-13T08:16:38.307 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-13T08:16:38.308 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-13T08:16:38.308 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-13T08:16:38.308 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-13T08:16:38.308 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-13T08:16:38.308 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "fsmap": { 2024-01-13T08:16:38.308 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T08:16:38.308 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-13T08:16:38.308 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-13T08:16:38.308 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-13T08:16:38.308 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-13T08:16:38.308 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "available": false, 2024-01-13T08:16:38.308 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-13T08:16:38.308 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "modules": [ 2024-01-13T08:16:38.309 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "iostat", 2024-01-13T08:16:38.309 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "nfs", 2024-01-13T08:16:38.309 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "restful" 2024-01-13T08:16:38.309 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: ], 2024-01-13T08:16:38.309 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "services": {} 2024-01-13T08:16:38.309 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-13T08:16:38.309 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "servicemap": { 2024-01-13T08:16:38.309 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T08:16:38.309 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "modified": "2024-01-13T08:16:31.543552+0000", 2024-01-13T08:16:38.309 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "services": {} 2024-01-13T08:16:38.309 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-13T08:16:38.309 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-13T08:16:38.309 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: } 2024-01-13T08:16:38.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:38 smithi099 conmon[33313]: audit 2024-01-13T08:16:38.299896+0000 2024-01-13T08:16:38.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:38 smithi099 conmon[33313]: mon.smithi099 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.99:0/497889256' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-13T08:16:38.863 INFO:teuthology.orchestra.run.smithi099.stderr:mgr not available, waiting (1/15)... 2024-01-13T08:16:41.401 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: 2024-01-13T08:16:41.402 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: { 2024-01-13T08:16:41.402 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "fsid": "f1ba87a2-b1eb-11ee-95ac-87774f69a715", 2024-01-13T08:16:41.402 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "health": { 2024-01-13T08:16:41.402 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-13T08:16:41.402 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "checks": {}, 2024-01-13T08:16:41.402 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "mutes": [] 2024-01-13T08:16:41.402 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-13T08:16:41.402 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-13T08:16:41.402 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "quorum": [ 2024-01-13T08:16:41.402 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: 0 2024-01-13T08:16:41.402 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: ], 2024-01-13T08:16:41.402 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-13T08:16:41.403 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "smithi099" 2024-01-13T08:16:41.403 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: ], 2024-01-13T08:16:41.403 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "quorum_age": 5, 2024-01-13T08:16:41.403 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "monmap": { 2024-01-13T08:16:41.403 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T08:16:41.403 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-13T08:16:41.403 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-13T08:16:41.403 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-13T08:16:41.403 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "osdmap": { 2024-01-13T08:16:41.403 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T08:16:41.403 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-13T08:16:41.403 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-13T08:16:41.403 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-13T08:16:41.403 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-13T08:16:41.404 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-13T08:16:41.404 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-13T08:16:41.404 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-13T08:16:41.404 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "pgmap": { 2024-01-13T08:16:41.404 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-13T08:16:41.404 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-13T08:16:41.404 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-13T08:16:41.404 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-13T08:16:41.404 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-13T08:16:41.404 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-13T08:16:41.404 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-13T08:16:41.404 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-13T08:16:41.404 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-13T08:16:41.405 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "fsmap": { 2024-01-13T08:16:41.405 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T08:16:41.405 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-13T08:16:41.405 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-13T08:16:41.405 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-13T08:16:41.405 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-13T08:16:41.405 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "available": false, 2024-01-13T08:16:41.405 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-13T08:16:41.405 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "modules": [ 2024-01-13T08:16:41.405 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "iostat", 2024-01-13T08:16:41.405 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "nfs", 2024-01-13T08:16:41.405 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "restful" 2024-01-13T08:16:41.405 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: ], 2024-01-13T08:16:41.406 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "services": {} 2024-01-13T08:16:41.406 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-13T08:16:41.406 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "servicemap": { 2024-01-13T08:16:41.406 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T08:16:41.406 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "modified": "2024-01-13T08:16:31.543552+0000", 2024-01-13T08:16:41.406 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "services": {} 2024-01-13T08:16:41.406 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-13T08:16:41.406 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-13T08:16:41.406 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: } 2024-01-13T08:16:41.987 INFO:teuthology.orchestra.run.smithi099.stderr:mgr not available, waiting (2/15)... 2024-01-13T08:16:42.201 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:41 smithi099 conmon[33313]: audit 2024-01-13T08:16:41.398849+0000 mon.smithi099 (mon.0) 2024-01-13T08:16:42.201 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:41 smithi099 conmon[33313]: 8 : audit [DBG] from='client.? 172.21.15.99:0/1666950871' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-13T08:16:43.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:43 smithi099 conmon[33313]: cluster 2024-01-13T08:16:43.187068+0000 mon.smithi099 (mon.0) 9 : cluster [INF] Activating manager daemon smithi099.oodult 2024-01-13T08:16:43.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:43 smithi099 conmon[33313]: cluster 2024-01-13T08:16:43.189640+0000 mon.smithi099 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi099.oodult(active, starting, since 0.00267636s) 2024-01-13T08:16:43.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:43 smithi099 conmon[33313]: audit 2024-01-13T08:16:43.190337+0000 mon.smithi099 (mon.0) 11 : audit [DBG] 2024-01-13T08:16:43.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:43 smithi099 conmon[33313]: from='mgr.14100 172.21.15.99:0/3029512605' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-13T08:16:43.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:43 smithi099 conmon[33313]: audit 2024-01-13T08:16:43.190540 2024-01-13T08:16:43.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:43 smithi099 conmon[33313]: +0000 mon.smithi099 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.99:0/3029512605' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-13T08:16:43.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:43 smithi099 conmon[33313]: audit 2024-01-13T08:16 2024-01-13T08:16:43.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:43 smithi099 conmon[33313]: :43.190715+0000 mon.smithi099 (mon.0) 13 2024-01-13T08:16:43.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:43 smithi099 conmon[33313]: : audit [DBG] from='mgr.14100 172.21.15.99:0/3029512605' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-13T08:16:43.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:43 smithi099 conmon[33313]: audit 2024-01-13T08:16:43. 2024-01-13T08:16:43.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:43 smithi099 conmon[33313]: 190877+0000 mon.smithi099 (mon.0) 14 : audit [DBG] 2024-01-13T08:16:43.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:43 smithi099 conmon[33313]: from='mgr.14100 172.21.15.99:0/3029512605' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-01-13T08:16:43.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:43 smithi099 conmon[33313]: audit 2024-01-13T08:16:43.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:43 smithi099 conmon[33313]: 2024-01-13T08:16:43.191068+0000 mon.smithi099 (mon.0 2024-01-13T08:16:43.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:43 smithi099 conmon[33313]: ) 15 : audit [DBG] from='mgr.14100 172.21.15.99:0/3029512605' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mgr metadata", "who": "smithi099.oodult", "id": "smithi099.oodult"}]: dispatch 2024-01-13T08:16:43.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:43 smithi099 conmon[33313]: cluster 2024-01-13T08:16 2024-01-13T08:16:43.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:43 smithi099 conmon[33313]: :43.204846+0000 mon.smithi099 (mon.0) 16 2024-01-13T08:16:43.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:43 smithi099 conmon[33313]: : cluster [INF] Manager daemon smithi099.oodult is now available 2024-01-13T08:16:43.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:43 smithi099 conmon[33313]: audit 2024-01-13T08:16:43 2024-01-13T08:16:43.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:43 smithi099 conmon[33313]: .216285+0000 mon.smithi099 (mon.0) 17 : audit 2024-01-13T08:16:43.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:43 smithi099 conmon[33313]: [INF] from='mgr.14100 172.21.15.99:0/3029512605' entity='mgr.smithi099.oodult' 2024-01-13T08:16:43.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:43 smithi099 conmon[33313]: audit 2024-01-13T08:16:43.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:43 smithi099 conmon[33313]: 2024-01-13T08:16:43.219400+0000 mon.smithi099 ( 2024-01-13T08:16:43.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:43 smithi099 conmon[33313]: mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.99:0/3029512605' entity='mgr.smithi099.oodult' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi099.oodult/mirror_snapshot_schedule"}]: dispatch 2024-01-13T08:16:43.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:43 smithi099 conmon[33313]: audit 2024-01-13 2024-01-13T08:16:43.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:43 smithi099 conmon[33313]: T08:16:43.222148+0000 mon.smithi099 (mon 2024-01-13T08:16:43.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:43 smithi099 conmon[33313]: .0) 19 : audit [INF] from='mgr.14100 172.21.15.99:0/3029512605' entity='mgr.smithi099.oodult' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi099.oodult/trash_purge_schedule"}]: dispatch 2024-01-13T08:16:43.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:43 smithi099 conmon[33313]: audit 2024-01-13T 2024-01-13T08:16:43.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:43 smithi099 conmon[33313]: 08:16:43.227080+0000 mon.smithi099 (mon. 2024-01-13T08:16:43.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:43 smithi099 conmon[33313]: 0) 20 : audit [INF] from='mgr.14100 172.21.15.99:0/3029512605' entity='mgr.smithi099.oodult' 2024-01-13T08:16:43.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:43 smithi099 conmon[33313]: audit 2024-01-13T08: 2024-01-13T08:16:43.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:43 smithi099 conmon[33313]: 16:43.232377+0000 mon.smithi099 (mon.0) 21 2024-01-13T08:16:43.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:43 smithi099 conmon[33313]: : audit [INF] from='mgr.14100 172.21.15.99:0/3029512605' entity='mgr.smithi099.oodult' 2024-01-13T08:16:44.649 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: 2024-01-13T08:16:44.649 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: { 2024-01-13T08:16:44.649 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "fsid": "f1ba87a2-b1eb-11ee-95ac-87774f69a715", 2024-01-13T08:16:44.649 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "health": { 2024-01-13T08:16:44.649 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-13T08:16:44.649 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "checks": {}, 2024-01-13T08:16:44.649 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "mutes": [] 2024-01-13T08:16:44.649 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-13T08:16:44.650 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-13T08:16:44.650 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "quorum": [ 2024-01-13T08:16:44.650 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: 0 2024-01-13T08:16:44.650 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: ], 2024-01-13T08:16:44.650 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-13T08:16:44.650 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "smithi099" 2024-01-13T08:16:44.650 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: ], 2024-01-13T08:16:44.650 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "quorum_age": 8, 2024-01-13T08:16:44.650 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "monmap": { 2024-01-13T08:16:44.650 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T08:16:44.650 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-13T08:16:44.650 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-13T08:16:44.650 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-13T08:16:44.650 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "osdmap": { 2024-01-13T08:16:44.650 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T08:16:44.650 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-13T08:16:44.651 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-13T08:16:44.651 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-13T08:16:44.651 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-13T08:16:44.651 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-13T08:16:44.651 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-13T08:16:44.651 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-13T08:16:44.651 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "pgmap": { 2024-01-13T08:16:44.651 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-13T08:16:44.651 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-13T08:16:44.651 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-13T08:16:44.651 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-13T08:16:44.651 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-13T08:16:44.651 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-13T08:16:44.651 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-13T08:16:44.652 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-13T08:16:44.652 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-13T08:16:44.652 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "fsmap": { 2024-01-13T08:16:44.652 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T08:16:44.652 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-13T08:16:44.652 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-13T08:16:44.652 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-13T08:16:44.652 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-13T08:16:44.652 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "available": true, 2024-01-13T08:16:44.652 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-13T08:16:44.652 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "modules": [ 2024-01-13T08:16:44.652 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "iostat", 2024-01-13T08:16:44.652 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "nfs", 2024-01-13T08:16:44.652 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "restful" 2024-01-13T08:16:44.652 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: ], 2024-01-13T08:16:44.653 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "services": {} 2024-01-13T08:16:44.653 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-13T08:16:44.653 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "servicemap": { 2024-01-13T08:16:44.653 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-13T08:16:44.653 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "modified": "2024-01-13T08:16:31.543552+0000", 2024-01-13T08:16:44.653 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "services": {} 2024-01-13T08:16:44.653 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: }, 2024-01-13T08:16:44.653 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-13T08:16:44.653 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: } 2024-01-13T08:16:45.145 INFO:teuthology.orchestra.run.smithi099.stderr:mgr is available 2024-01-13T08:16:45.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:45 smithi099 conmon[33313]: cluster 2024-01-13T08:16:44.192309+0000 mon.smithi099 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi099.oodult(active, since 1.00535s) 2024-01-13T08:16:45.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:45 smithi099 conmon[33313]: audit 2024-01-13T08:16:45.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:45 smithi099 conmon[33313]: 2024-01-13T08:16:44.646653+0000 mon.smithi099 (mon.0) 23 : audit [DBG] from='client.? 172.21.15.99:0/3418009811' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-13T08:16:45.710 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: 2024-01-13T08:16:45.710 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: [global] 2024-01-13T08:16:45.710 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: fsid = f1ba87a2-b1eb-11ee-95ac-87774f69a715 2024-01-13T08:16:45.710 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-01-13T08:16:45.710 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-01-13T08:16:45.710 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-01-13T08:16:45.710 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-01-13T08:16:45.710 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: 2024-01-13T08:16:45.710 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: [mgr] 2024-01-13T08:16:45.711 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-01-13T08:16:45.711 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: 2024-01-13T08:16:45.711 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: [osd] 2024-01-13T08:16:45.711 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-01-13T08:16:45.711 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-01-13T08:16:46.216 INFO:teuthology.orchestra.run.smithi099.stderr:Enabling cephadm module... 2024-01-13T08:16:46.437 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:46 smithi099 conmon[33313]: audit 2024-01-13T08:16:45.711369+0000 mon.smithi099 (mon.0) 2024-01-13T08:16:46.438 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:46 smithi099 conmon[33313]: 24 : audit [INF] from='client.? 172.21.15.99:0/2107714089' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-13T08:16:47.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:47 smithi099 conmon[33313]: cluster 2024-01-13T08:16:46.193459+0000 mon.smithi099 (mon.0) 25 : cluster [DBG] mgrmap e4: smithi099.oodult(active, since 3s) 2024-01-13T08:16:47.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:47 smithi099 conmon[33313]: audit 2024-01- 2024-01-13T08:16:47.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:47 smithi099 conmon[33313]: 13T08:16:46.860326+0000 mon.smithi099 (mon.0) 26 : audit [INF] from='client.? 172.21.15.99:0/2930588846' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-01-13T08:16:48.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:48 smithi099 conmon[33313]: audit 2024-01-13T08:16:47.194977+0000 mon.smithi099 (mon.0) 27 : 2024-01-13T08:16:48.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:48 smithi099 conmon[33313]: audit [INF] from='client.? 172.21.15.99:0/2930588846' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-01-13T08:16:48.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:48 smithi099 conmon[33313]: cluster 2024-01-13T08:16:47.195030+0000 mon.smithi099 (mon.0) 28 : cluster [DBG] mgrmap e5: smithi099.oodult(active, since 4s) 2024-01-13T08:16:48.389 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: { 2024-01-13T08:16:48.389 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "epoch": 5, 2024-01-13T08:16:48.389 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "available": true, 2024-01-13T08:16:48.389 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "active_name": "smithi099.oodult", 2024-01-13T08:16:48.389 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_standby": 0 2024-01-13T08:16:48.390 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: } 2024-01-13T08:16:48.877 INFO:teuthology.orchestra.run.smithi099.stderr:Waiting for the mgr to restart... 2024-01-13T08:16:48.877 INFO:teuthology.orchestra.run.smithi099.stderr:Waiting for mgr epoch 5... 2024-01-13T08:16:49.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:49 smithi099 conmon[33313]: audit 2024-01-13T08:16:48.391882 2024-01-13T08:16:49.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:49 smithi099 conmon[33313]: +0000 mon.smithi099 (mon.0) 29 : audit [DBG] from='client.? 172.21.15.99:0/1949992485' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-13T08:16:52.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:52 smithi099 conmon[33313]: cluster 2024-01-13T08:16:52.572097+0000 mon.smithi099 (mon.0) 30 : cluster 2024-01-13T08:16:52.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:52 smithi099 conmon[33313]: [INF] Active manager daemon smithi099.oodult restarted 2024-01-13T08:16:52.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:52 smithi099 conmon[33313]: cluster 2024-01-13T08:16:52.572317+0000 mon.smithi099 (mon.0) 31 : cluster [INF] Activating manager daemon smithi099.oodult 2024-01-13T08:16:52.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:52 smithi099 conmon[33313]: cluster 2024-01-13T08:16:52.575332+0000 mon.smithi099 (mon.0 2024-01-13T08:16:52.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:52 smithi099 conmon[33313]: ) 32 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-01-13T08:16:52.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:52 smithi099 conmon[33313]: cluster 2024-01-13T08:16:52.575439+0000 2024-01-13T08:16:52.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:52 smithi099 conmon[33313]: mon.smithi099 (mon.0) 33 : cluster [DBG] mgrmap e6: smithi099.oodult(active, starting, since 0.00319981s) 2024-01-13T08:16:52.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:52 smithi099 conmon[33313]: audit 2024-01-13T 2024-01-13T08:16:52.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:52 smithi099 conmon[33313]: 08:16:52.576379+0000 mon.smithi099 (mon.0) 34 : audit 2024-01-13T08:16:52.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:52 smithi099 conmon[33313]: [DBG] from='mgr.14116 172.21.15.99:0/1899221123' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-01-13T08:16:52.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:52 smithi099 conmon[33313]: audit 2024-01-13T08:16: 2024-01-13T08:16:52.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:52 smithi099 conmon[33313]: 52.576963+0000 mon.smithi099 (mon.0) 35 : audit 2024-01-13T08:16:52.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:52 smithi099 conmon[33313]: [DBG] from='mgr.14116 172.21.15.99:0/1899221123' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mgr metadata", "who": "smithi099.oodult", "id": "smithi099.oodult"}]: dispatch 2024-01-13T08:16:52.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:52 smithi099 conmon[33313]: audit 2024-01 2024-01-13T08:16:52.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:52 smithi099 conmon[33313]: -13T08:16:52.577582+0000 mon.smithi099 (mon.0 2024-01-13T08:16:52.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:52 smithi099 conmon[33313]: ) 36 : audit [DBG] from='mgr.14116 172.21.15.99:0/1899221123' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-13T08:16:52.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:52 smithi099 conmon[33313]: audit 2024-01 2024-01-13T08:16:52.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:52 smithi099 conmon[33313]: -13T08:16:52.577749+0000 mon.smithi099 (mon 2024-01-13T08:16:52.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:52 smithi099 conmon[33313]: .0) 37 : audit [DBG] from='mgr.14116 172.21.15.99:0/1899221123' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-13T08:16:52.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:52 smithi099 conmon[33313]: 2024-01-13T08:16:52.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:52 smithi099 conmon[33313]: audit 2024-01-13T08 2024-01-13T08:16:52.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:52 smithi099 conmon[33313]: :16:52.577870+0000 mon.smithi099 (mon.0) 38 2024-01-13T08:16:52.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:52 smithi099 conmon[33313]: : audit [DBG] from='mgr.14116 172.21.15.99:0/1899221123' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-13T08:16:52.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:52 smithi099 conmon[33313]: cluster 2024-01 2024-01-13T08:16:52.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:52 smithi099 conmon[33313]: -13T08:16:52.591589+0000 mon.smithi099 (mon 2024-01-13T08:16:52.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:52 smithi099 conmon[33313]: .0) 39 : cluster [INF] Manager daemon smithi099.oodult is now available 2024-01-13T08:16:52.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:52 smithi099 conmon[33313]: audit 2024- 2024-01-13T08:16:52.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:52 smithi099 conmon[33313]: 01-13T08:16:52.603510+0000 mon.smithi099 (mon. 2024-01-13T08:16:52.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:52 smithi099 conmon[33313]: 0) 40 : audit [INF] from='mgr.14116 172.21.15.99:0/1899221123' entity='mgr.smithi099.oodult' 2024-01-13T08:16:53.581 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: { 2024-01-13T08:16:53.581 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-01-13T08:16:53.582 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "initialized": true 2024-01-13T08:16:53.582 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: } 2024-01-13T08:16:53.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:53 smithi099 conmon[33313]: cephadm 2024-01-13T08:16:52.600985+0000 mgr.smithi099.oodult (mgr.14116) 1 : cephadm [INF] Found migration_current of "None". Setting to last migration. 2024-01-13T08:16:53.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:53 smithi099 conmon[33313]: audit 2024-01-13T08:16:53.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:53 smithi099 conmon[33313]: 2024-01-13T08:16:52.687827+0000 mon.smithi099 (mon.0) 41 : audit [INF] from='mgr.14116 172.21.15.99:0/1899221123' entity='mgr.smithi099.oodult' 2024-01-13T08:16:53.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:53 smithi099 conmon[33313]: audit 2024-01-13T08:16:52.688818+0000 mon.smithi099 (mon.0) 42 : audit [DBG] from='mgr.14116 172.21.15.99:0/1899221123' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:16:53.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:53 smithi099 conmon[33313]: audit 2024-01-13T08:16:52.690890+0000 mon.smithi099 (mon.0 2024-01-13T08:16:53.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:53 smithi099 conmon[33313]: ) 43 : audit [DBG] from='mgr.14116 172.21.15.99:0/1899221123' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:16:53.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:53 smithi099 conmon[33313]: audit 2024-01-13T08:16:52.693015+0000 mon.smithi099 2024-01-13T08:16:53.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:53 smithi099 conmon[33313]: (mon.0) 44 : audit [DBG] from='mgr.14116 172.21.15.99:0/1899221123' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:16:53.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:53 smithi099 conmon[33313]: audit 2024-01 2024-01-13T08:16:53.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:53 smithi099 conmon[33313]: -13T08:16:52.706325+0000 mon.smithi099 ( 2024-01-13T08:16:53.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:53 smithi099 conmon[33313]: mon.0) 45 : audit [INF] from='mgr.14116 172.21.15.99:0/1899221123' entity='mgr.smithi099.oodult' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi099.oodult/mirror_snapshot_schedule"}]: dispatch 2024-01-13T08:16:53.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:53 smithi099 conmon[33313]: audit 2024-01-13T08 2024-01-13T08:16:53.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:53 smithi099 conmon[33313]: :16:52.708742+0000 mon.smithi099 (mon.0) 2024-01-13T08:16:53.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:53 smithi099 conmon[33313]: 46 : audit [INF] from='mgr.14116 172.21.15.99:0/1899221123' entity='mgr.smithi099.oodult' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi099.oodult/trash_purge_schedule"}]: dispatch 2024-01-13T08:16:53.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:53 smithi099 conmon[33313]: cluster 2024-01-13T08:16 2024-01-13T08:16:53.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:53 smithi099 conmon[33313]: :53.577935+0000 mon.smithi099 (mon.0) 47 : 2024-01-13T08:16:53.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:53 smithi099 conmon[33313]: cluster [DBG] mgrmap e7: smithi099.oodult(active, since 1.00569s) 2024-01-13T08:16:54.085 INFO:teuthology.orchestra.run.smithi099.stderr:mgr epoch 5 is available 2024-01-13T08:16:54.085 INFO:teuthology.orchestra.run.smithi099.stderr:Setting orchestrator backend to cephadm... 2024-01-13T08:16:54.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:54 smithi099 conmon[33313]: audit 2024-01-13T08:16:53.577077+0000 mgr.smithi099.oodult (mgr.14116) 2 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-13T08:16:54.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:54 smithi099 conmon[33313]: audit 2024-01-13T08:16:53.584183+0000 mgr.smithi099.oodult (mgr.14116) 3 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-13T08:16:54.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:54 smithi099 conmon[33313]: audit 2024-01-13T08:16:54.667524+0000 mon.smithi099 (mon.0) 48 : audit [INF] from='mgr.14116 172.21.15.99:0/1899221123' entity='mgr.smithi099.oodult' 2024-01-13T08:16:55.711 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: value unchanged 2024-01-13T08:16:55.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:55 smithi099 conmon[33313]: audit 2024-01-13T08:16:54.663690+0000 2024-01-13T08:16:55.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:55 smithi099 conmon[33313]: mgr.smithi099.oodult (mgr.14116) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T08:16:55.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:55 smithi099 conmon[33313]: cluster 2024-01-13T08:16:54.691578+0000 mon.smithi099 (mon.0) 49 : cluster [DBG] mgrmap e8: smithi099.oodult(active, since 2s) 2024-01-13T08:16:55.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:55 smithi099 conmon[33313]: audit 2024-01-13T08:16:54.716738+0000 mon.smithi099 (mon.0) 50 : audit [DBG] from='mgr.14116 172.21.15.99:0/1899221123' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:16:56.242 INFO:teuthology.orchestra.run.smithi099.stderr:Generating ssh key... 2024-01-13T08:16:57.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:56 smithi099 conmon[33313]: audit 2024-01-13T08:16:55.713794 2024-01-13T08:16:57.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:56 smithi099 conmon[33313]: +0000 mgr.smithi099.oodult (mgr.14116) 5 : audit [DBG] from='client.14128 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T08:16:58.159 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcirLo7lt3Tk+l/5ZacdAUCgBHrPhbFNeMS+qImMwzkCLvQz7gNQ6KQqyOjoHoW0BQzk2DTN4LP3pXibU7egufaBwV3kbI2xAIR4NxHPe5JhXLarAN6A698mtLwzsaDV/82PTu5X3Z+k6xuSZW/Q5Rp55jhcl2wYBcAPPozvXQwtFLUQDlsmr/BjEUzZu46Tgw2DZrZqowEzXp3KTYTsrTj27YuR+OsVO1YdqMlIhCWzB6H9FdSW7w9oKZs/swqyv5MmYFWV0cGpK3j68J/P9M6iyxRcOtBtY6vt2Srql1zJ+S987y8o/yPeaGL9WV5S/KqcPSOO5Tzx8xt34kPJx7TF3jlMH8LjiB//ruZ2d2If96mazrfB+8oxD/5ncwYB00jgYmKRoYWQq6HkBx1q1YdN5NFa05PzKGO69/XePx4QNvmwUL/Wn1KVNNGlEpJTkZm3It+OSs91V90wYa4X8J2tIVSCWzPZc4odLPC2zGxeclYkYDhx9tRArUAJ6vgqU= ceph-f1ba87a2-b1eb-11ee-95ac-87774f69a715 2024-01-13T08:16:58.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:58 smithi099 conmon[33313]: audit 2024-01-13T08:16:56.815462+0000 mgr.smithi099.oodult (mgr.14116) 6 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T08:16:58.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:58 smithi099 conmon[33313]: cephadm 2024-01-13T08:16:56.815671+0000 mgr.smithi099.oodult (mgr.14116) 7 : cephadm [INF] Generating ssh key... 2024-01-13T08:16:58.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:58 smithi099 conmon[33313]: audit 2024-01-13T08:16:57.111850+0000 mon.smithi099 (mon.0) 51 : audit [INF] from='mgr.14116 172.21.15.99:0/1899221123' entity='mgr.smithi099.oodult' 2024-01-13T08:16:58.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:58 smithi099 conmon[33313]: audit 2024-01-13T08:16:57.113561+0000 mon.smithi099 (mon.0) 52 : audit [INF] from='mgr.14116 172.21.15.99:0/1899221123' entity='mgr.smithi099.oodult' 2024-01-13T08:16:58.651 INFO:teuthology.orchestra.run.smithi099.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-01-13T08:16:58.651 INFO:teuthology.orchestra.run.smithi099.stderr:Adding key to root@localhost authorized_keys... 2024-01-13T08:16:58.652 INFO:teuthology.orchestra.run.smithi099.stderr:Adding host smithi099... 2024-01-13T08:16:59.377 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:59 smithi099 conmon[33313]: audit 2024-01-13T08:16:58.162532+0000 mgr.smithi099.oodult (mgr.14116) 2024-01-13T08:16:59.377 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:16:59 smithi099 conmon[33313]: 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T08:16:59.849 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: Added host 'smithi099' with addr '172.21.15.99' 2024-01-13T08:17:00.358 INFO:teuthology.orchestra.run.smithi099.stderr:Deploying mon service with default placement... 2024-01-13T08:17:01.011 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: Scheduled mon update... 2024-01-13T08:17:01.212 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:00 smithi099 conmon[33313]: audit 2024-01-13T08:16:59.239852+0000 mgr.smithi099.oodult (mgr.14116) 9 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi099", "addr": "172.21.15.99", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T08:17:01.212 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:00 smithi099 conmon[33313]: audit 2024-01-13T08:16:59.850849+0000 mon.smithi099 (mon.0) 53 : audit [INF] from='mgr.14116 172.21.15.99:0/1899221123' entity='mgr.smithi099.oodult' 2024-01-13T08:17:01.212 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:00 smithi099 conmon[33313]: cephadm 2024-01-13T08:16:59.851114+0000 mgr.smithi099.oodult (mgr.14116) 10 : cephadm [INF] Added host smithi099 2024-01-13T08:17:01.213 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:00 smithi099 conmon[33313]: audit 2024-01-13T08:16:59.896642+0000 mon.smithi099 (mon.0) 54 : audit [DBG] from='mgr.14116 172.21.15.99:0/1899221123' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:17:01.883 INFO:teuthology.orchestra.run.smithi099.stderr:Deploying mgr service with default placement... 2024-01-13T08:17:02.149 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:02 smithi099 conmon[33313]: audit 2024-01-13T08:17:01.010626+0000 mgr.smithi099.oodult (mgr.14116) 11 : 2024-01-13T08:17:02.149 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:02 smithi099 conmon[33313]: audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T08:17:02.149 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:02 smithi099 conmon[33313]: cephadm 2024-01-13T08:17:01.011757+0000 mgr.smithi099.oodult (mgr.14116) 12 : cephadm [INF] Saving service mon spec with placement count:5 2024-01-13T08:17:02.150 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:02 smithi099 conmon[33313]: audit 2024-01-13T08:17:01.013950+0000 mon.smithi099 (mon.0) 55 : audit [INF] from='mgr.14116 172.21.15.99:0/1899221123' entity='mgr.smithi099.oodult' 2024-01-13T08:17:02.150 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:02 smithi099 conmon[33313]: audit 2024-01-13T08:17:01.857838+0000 mon.smithi099 (mon.0) 56 : audit [INF] from='mgr.14116 172.21.15.99:0/1899221123' entity='mgr.smithi099.oodult' 2024-01-13T08:17:02.599 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-01-13T08:17:03.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:03 smithi099 conmon[33313]: audit 2024-01-13T08:17:02.188793+0000 mon.smithi099 (mon.0) 57 : audit [INF] 2024-01-13T08:17:03.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:03 smithi099 conmon[33313]: from='mgr.14116 172.21.15.99:0/1899221123' entity='mgr.smithi099.oodult' 2024-01-13T08:17:03.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:03 smithi099 conmon[33313]: audit 2024-01-13T08:17:02.602048+0000 mon.smithi099 (mon.0) 58 : audit [INF] from='mgr.14116 172.21.15.99:0/1899221123' entity='mgr.smithi099.oodult' 2024-01-13T08:17:03.407 INFO:teuthology.orchestra.run.smithi099.stderr:Deploying crash service with default placement... 2024-01-13T08:17:04.057 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: Scheduled crash update... 2024-01-13T08:17:04.415 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:04 smithi099 conmon[33313]: audit 2024-01-13T08:17:02.598776+0000 mgr.smithi099.oodult (mgr.14116) 13 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T08:17:04.415 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:04 smithi099 conmon[33313]: cephadm 2024-01-13T08:17:02.599686+0000 mgr.smithi099.oodult (mgr.14116) 14 : cephadm [INF] Saving service mgr spec with placement count:2 2024-01-13T08:17:04.415 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:04 smithi099 conmon[33313]: audit 2024-01-13T08:17:03.481394+0000 mon.smithi099 (mon.0) 59 : audit [INF] from='mgr.14116 172.21.15.99:0/1899221123' entity='mgr.smithi099.oodult' 2024-01-13T08:17:04.415 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:04 smithi099 conmon[33313]: audit 2024-01-13T08:17:04.060105+0000 mon.smithi099 (mon.0) 60 : audit [INF] from='mgr.14116 172.21.15.99:0/1899221123' entity='mgr.smithi099.oodult' 2024-01-13T08:17:05.190 INFO:teuthology.orchestra.run.smithi099.stderr:Deploying prometheus service with default placement... 2024-01-13T08:17:05.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:05 smithi099 conmon[33313]: audit 2024-01-13T08:17:04.056958+0000 mgr.smithi099.oodult (mgr.14116) 15 : audit [DBG] 2024-01-13T08:17:05.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:05 smithi099 conmon[33313]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T08:17:05.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:05 smithi099 conmon[33313]: cephadm 2024-01-13T08:17:04.057889+0000 mgr.smithi099.oodult (mgr.14116) 16 : cephadm [INF] Saving service crash spec with placement * 2024-01-13T08:17:05.782 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-01-13T08:17:06.356 INFO:teuthology.orchestra.run.smithi099.stderr:Deploying grafana service with default placement... 2024-01-13T08:17:06.946 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-01-13T08:17:07.122 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:06 smithi099 conmon[33313]: audit 2024-01-13T08:17:05.782438+0000 mgr.smithi099.oodult (mgr.14116) 17 : audit 2024-01-13T08:17:07.122 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:06 smithi099 conmon[33313]: [DBG] from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T08:17:07.122 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:06 smithi099 conmon[33313]: cephadm 2024-01-13T08:17:05.783297+0000 mgr.smithi099.oodult (mgr.14116) 18 : cephadm [INF] Saving service prometheus spec with placement count:1 2024-01-13T08:17:07.122 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:06 smithi099 conmon[33313]: audit 2024-01-13T08:17:05.785565+0000 mon.smithi099 (mon.0) 61 : audit [INF] from='mgr.14116 172.21.15.99:0/1899221123' entity='mgr.smithi099.oodult' 2024-01-13T08:17:07.481 INFO:teuthology.orchestra.run.smithi099.stderr:Deploying node-exporter service with default placement... 2024-01-13T08:17:08.094 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-01-13T08:17:08.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:07 smithi099 conmon[33313]: audit 2024-01-13T08:17:06.945058+0000 2024-01-13T08:17:08.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:07 smithi099 conmon[33313]: mgr.smithi099.oodult (mgr.14116) 19 : audit [DBG] from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T08:17:08.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:07 smithi099 conmon[33313]: cephadm 2024-01-13T08:17:06.945884+0000 mgr.smithi099.oodult (mgr.14116) 20 : cephadm [INF] Saving service grafana spec with placement count:1 2024-01-13T08:17:08.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:07 smithi099 conmon[33313]: audit 2024-01-13T08:17:06.948702+0000 mon.smithi099 (mon.0) 62 : audit [INF] from='mgr.14116 172.21.15.99:0/1899221123' entity='mgr.smithi099.oodult' 2024-01-13T08:17:08.631 INFO:teuthology.orchestra.run.smithi099.stderr:Deploying alertmanager service with default placement... 2024-01-13T08:17:09.197 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-01-13T08:17:09.363 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:09 smithi099 conmon[33313]: audit 2024-01-13T08:17:08.092684+0000 mgr.smithi099.oodult (mgr.14116) 21 : audit [DBG] from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T08:17:09.363 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:09 smithi099 conmon[33313]: cephadm 2024-01-13T08:17:08.093540+0000 mgr.smithi099.oodult (mgr.14116) 22 : cephadm [INF] Saving service node-exporter spec with placement * 2024-01-13T08:17:09.363 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:09 smithi099 conmon[33313]: audit 2024-01-13T08:17:08.096466+0000 mon.smithi099 (mon.0) 63 : audit [INF] from='mgr.14116 172.21.15.99:0/1899221123' entity='mgr.smithi099.oodult' 2024-01-13T08:17:10.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:10 smithi099 conmon[33313]: audit 2024-01-13T08:17:09.197015+0000 mgr.smithi099.oodult (mgr.14116) 23 : audit [DBG] from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T08:17:10.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:10 smithi099 conmon[33313]: cephadm 2024-01-13T08:17:09.197820+0000 mgr.smithi099.oodult (mgr.14116) 24 : cephadm [INF] Saving service alertmanager spec with placement count:1 2024-01-13T08:17:10.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:10 smithi099 conmon[33313]: audit 2024-01-13T08:17:09.199986+0000 mon.smithi099 (mon.0) 64 : audit [INF] from='mgr.14116 172.21.15.99:0/1899221123' entity='mgr.smithi099.oodult' 2024-01-13T08:17:11.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:11 smithi099 conmon[33313]: audit 2024-01-13T08:17:10 2024-01-13T08:17:11.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:11 smithi099 conmon[33313]: .351075+0000 mon.smithi099 (mon.0) 65 : audit [INF] from='client.? 172.21.15.99:0/2011432020' entity='client.admin' 2024-01-13T08:17:11.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:11 smithi099 conmon[33313]: audit 2024-01-13T08:17:11. 2024-01-13T08:17:11.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:11 smithi099 conmon[33313]: 346876+0000 mon.smithi099 (mon.0) 66 : audit [INF] from='client.? 172.21.15.99:0/1735371335' entity='client.admin' 2024-01-13T08:17:11.847 INFO:teuthology.orchestra.run.smithi099.stderr:Enabling the dashboard module... 2024-01-13T08:17:12.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:12 smithi099 conmon[33313]: audit 2024-01-13T08:17:12.498284+0000 mon.smithi099 (mon.0) 67 2024-01-13T08:17:12.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:12 smithi099 conmon[33313]: : audit [INF] from='client.? 172.21.15.99:0/3563117540' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-01-13T08:17:13.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:13 smithi099 conmon[33313]: audit 2024-01-13T08:17:12.551572+0000 mon.smithi099 (mon.0) 2024-01-13T08:17:13.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:13 smithi099 conmon[33313]: 68 : audit [INF] from='client.? 172.21.15.99:0/3563117540' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-01-13T08:17:13.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:13 smithi099 conmon[33313]: cluster 2024-01-13T08:17:12.551640+0000 mon.smithi099 (mon.0) 69 : cluster [DBG] mgrmap e9: smithi099.oodult(active, since 19s) 2024-01-13T08:17:13.785 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: { 2024-01-13T08:17:13.785 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "epoch": 9, 2024-01-13T08:17:13.785 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "available": true, 2024-01-13T08:17:13.786 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "active_name": "smithi099.oodult", 2024-01-13T08:17:13.786 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "num_standby": 0 2024-01-13T08:17:13.786 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: } 2024-01-13T08:17:14.330 INFO:teuthology.orchestra.run.smithi099.stderr:Waiting for the mgr to restart... 2024-01-13T08:17:14.330 INFO:teuthology.orchestra.run.smithi099.stderr:Waiting for mgr epoch 9... 2024-01-13T08:17:14.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:14 smithi099 conmon[33313]: audit 2024-01-13T08:17:13.788233+0000 mon.smithi099 (mon.0) 70 : audit 2024-01-13T08:17:14.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:14 smithi099 conmon[33313]: [DBG] from='client.? 172.21.15.99:0/3650199671' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-13T08:17:18.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:18 smithi099 conmon[33313]: cluster 2024-01-13T08:17:17.964133+0000 mon.smithi099 (mon.0) 71 : cluster [INF] Active manager daemon smithi099.oodult restarted 2024-01-13T08:17:18.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:18 smithi099 conmon[33313]: cluster 2024-01-13T08:17:17.964351+0000 mon.smithi099 (mon.0) 72 : cluster [INF] Activating manager daemon smithi099.oodult 2024-01-13T08:17:18.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:18 smithi099 conmon[33313]: cluster 2024-01-13T08:17:18.233 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:18 smithi099 conmon[33313]: 2024-01-13T08:17:17.967101 2024-01-13T08:17:18.233 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:18 smithi099 conmon[33313]: +0000 mon.smithi099 (mon.0) 73 : 2024-01-13T08:17:18.233 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:18 smithi099 conmon[33313]: cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-01-13T08:17:18.233 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:18 smithi099 conmon[33313]: cluster 2024-01-13T08:17:18.233 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:18 smithi099 conmon[33313]: 2024-01-13T08:17 2024-01-13T08:17:18.233 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:18 smithi099 conmon[33313]: :17.967217+0000 mon.smithi099 (mon 2024-01-13T08:17:18.233 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:18 smithi099 conmon[33313]: .0) 74 : cluster [DBG] mgrmap e10: smithi099.oodult(active, starting, since 0.00294892s) 2024-01-13T08:17:18.233 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:18 smithi099 conmon[33313]: 2024-01-13T08:17:18.233 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:18 smithi099 conmon[33313]: audit 2024-01- 2024-01-13T08:17:18.233 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:18 smithi099 conmon[33313]: 13T08:17:17.968185 2024-01-13T08:17:18.233 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:18 smithi099 conmon[33313]: +0000 mon.smithi099 (mon.0) 75 2024-01-13T08:17:18.233 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:18 smithi099 conmon[33313]: : audit [DBG] from='mgr.14156 172.21.15.99:0/2715542913' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-01-13T08:17:18.233 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:18 smithi099 conmon[33313]: audit 2024-01-13T08:17:18.233 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:18 smithi099 conmon[33313]: 2024-01-13T08:17 2024-01-13T08:17:18.233 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:18 smithi099 conmon[33313]: :17.968810+0000 mon.smithi099 ( 2024-01-13T08:17:18.234 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:18 smithi099 conmon[33313]: mon.0) 76 : audit [DBG] 2024-01-13T08:17:18.234 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:18 smithi099 conmon[33313]: from='mgr.14156 172.21.15.99:0/2715542913' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mgr metadata", "who": "smithi099.oodult", "id": "smithi099.oodult"}]: dispatch 2024-01-13T08:17:18.234 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:18 smithi099 conmon[33313]: audit 2024-01- 2024-01-13T08:17:18.234 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:18 smithi099 conmon[33313]: 13T08:17:17.969612 2024-01-13T08:17:18.234 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:18 smithi099 conmon[33313]: +0000 mon.smithi099 (mon.0) 2024-01-13T08:17:18.234 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:18 smithi099 conmon[33313]: 77 : audit [DBG] from='mgr.14156 172.21.15.99:0/2715542913' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-13T08:17:18.234 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:18 smithi099 conmon[33313]: audit 2024- 2024-01-13T08:17:18.234 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:18 smithi099 conmon[33313]: 01-13T08:17:17.969769 2024-01-13T08:17:18.234 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:18 smithi099 conmon[33313]: +0000 mon.smithi099 (mon.0) 78 2024-01-13T08:17:18.234 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:18 smithi099 conmon[33313]: : audit [DBG] from='mgr.14156 172.21.15.99:0/2715542913' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-13T08:17:18.234 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:18 smithi099 conmon[33313]: audit 2024-01-13T08:17:18.234 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:18 smithi099 conmon[33313]: 2024-01-13T08:17 2024-01-13T08:17:18.234 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:18 smithi099 conmon[33313]: :17.969910+0000 mon.smithi099 (mon 2024-01-13T08:17:18.234 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:18 smithi099 conmon[33313]: .0) 79 : audit [DBG] 2024-01-13T08:17:18.235 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:18 smithi099 conmon[33313]: from='mgr.14156 172.21.15.99:0/2715542913' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-13T08:17:18.235 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:18 smithi099 conmon[33313]: cluster 2024-01 2024-01-13T08:17:18.235 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:18 smithi099 conmon[33313]: -13T08:17:17. 2024-01-13T08:17:18.235 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:18 smithi099 conmon[33313]: 984715+0000 mon.smithi099 (mon.0) 2024-01-13T08:17:18.235 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:18 smithi099 conmon[33313]: 80 : cluster [INF] Manager daemon smithi099.oodult is now available 2024-01-13T08:17:18.972 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: { 2024-01-13T08:17:18.972 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-01-13T08:17:18.972 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: "initialized": true 2024-01-13T08:17:18.972 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: } 2024-01-13T08:17:19.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:19 smithi099 conmon[33313]: audit 2024-01-13T08:17:18.079582+0000 mon.smithi099 (mon.0) 81 : audit [DBG] from='mgr.14156 172.21.15.99:0/2715542913' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:17:19.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:19 smithi099 conmon[33313]: audit 2024-01-13T08:17:18.082233+0000 mon.smithi099 (mon.0) 82 : audit [DBG] from='mgr.14156 172.21.15.99:0/2715542913' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:17:19.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:19 smithi099 conmon[33313]: audit 2024-01-13T08:17:18.106490+0000 mon.smithi099 (mon.0) 83 : audit [INF] from='mgr.14156 172.21.15.99:0/2715542913' entity='mgr.smithi099.oodult' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi099.oodult/mirror_snapshot_schedule"}]: dispatch 2024-01-13T08:17:19.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:19 smithi099 conmon[33313]: audit 2024-01-13T08:17:18.109329+0000 mon.smithi099 (mon.0) 84 : audit [INF] from='mgr.14156 172.21.15.99:0/2715542913' entity='mgr.smithi099.oodult' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi099.oodult/trash_purge_schedule"}]: dispatch 2024-01-13T08:17:19.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:19 smithi099 conmon[33313]: cluster 2024-01-13T08:17:18.969126+0000 mon.smithi099 (mon.0) 85 : cluster [DBG] mgrmap e11: smithi099.oodult(active, since 1.00486s) 2024-01-13T08:17:19.570 INFO:teuthology.orchestra.run.smithi099.stderr:mgr epoch 9 is available 2024-01-13T08:17:19.570 INFO:teuthology.orchestra.run.smithi099.stderr:Generating a dashboard self-signed certificate... 2024-01-13T08:17:20.211 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:20 smithi099 conmon[33313]: audit 2024-01-13T08:17:18.968580+0000 mgr.smithi099.oodult (mgr.14156) 1 : audit [DBG] from='client.14160 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-13T08:17:20.211 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:20 smithi099 conmon[33313]: audit 2024-01-13T08:17:18.974732+0000 mgr.smithi099.oodult (mgr.14156) 2 : audit [DBG] from='client.14160 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-13T08:17:20.211 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:20 smithi099 conmon[33313]: audit 2024-01-13T08:17:19.131068+0000 mon.smithi099 (mon.0) 86 : audit [INF] from='mgr.14156 172.21.15.99:0/2715542913' entity='mgr.smithi099.oodult' 2024-01-13T08:17:20.354 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: Self-signed certificate created 2024-01-13T08:17:21.003 INFO:teuthology.orchestra.run.smithi099.stderr:Creating initial admin user... 2024-01-13T08:17:21.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:21 smithi099 conmon[33313]: cluster 2024-01-13T08:17:20.132585+0000 mon.smithi099 (mon.0) 2024-01-13T08:17:21.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:21 smithi099 conmon[33313]: 87 : cluster [DBG] mgrmap e12: smithi099.oodult(active, since 2s) 2024-01-13T08:17:21.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:21 smithi099 conmon[33313]: audit 2024-01-13T08:17:20.197394+0000 mgr.smithi099.oodult (mgr.14156) 3 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T08:17:21.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:21 smithi099 conmon[33313]: audit 2024-01-13T08:17:20.354577+0000 mon.smithi099 (mon.0) 88 : audit [INF] from='mgr.14156 172.21.15.99:0/2715542913' entity='mgr.smithi099.oodult' 2024-01-13T08:17:21.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:21 smithi099 conmon[33313]: audit 2024-01-13T08:17:20.356434+0000 mon.smithi099 (mon.0) 89 : audit [INF] from='mgr.14156 172.21.15.99:0/2715542913' entity='mgr.smithi099.oodult' 2024-01-13T08:17:21.838 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$rwQU2afhv2COd11dnKzuN.mJ21LPQa9xa2zBRZKM/9cCDIb0869F.", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1705133841, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-01-13T08:17:22.403 INFO:teuthology.orchestra.run.smithi099.stderr:Fetching dashboard port number... 2024-01-13T08:17:23.000 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: 8443 2024-01-13T08:17:23.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:22 smithi099 conmon[33313]: audit 2024-01-13T08:17:21.626066+0000 mgr.smithi099.oodult (mgr.14156) 4 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T08:17:23.168 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:22 smithi099 conmon[33313]: audit 2024-01-13T08:17:23.169 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:22 smithi099 conmon[33313]: 2024-01-13T08:17:21.841580+0000 mon.smithi099 (mon.0) 90 : audit [INF] from='mgr.14156 172.21.15.99:0/2715542913' entity='mgr.smithi099.oodult' 2024-01-13T08:17:23.595 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: disabled 2024-01-13T08:17:23.602 INFO:teuthology.orchestra.run.smithi099.stderr:systemctl: inactive 2024-01-13T08:17:23.603 INFO:teuthology.orchestra.run.smithi099.stderr:firewalld.service is not enabled 2024-01-13T08:17:23.603 INFO:teuthology.orchestra.run.smithi099.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-01-13T08:17:23.605 INFO:teuthology.orchestra.run.smithi099.stderr:Ceph Dashboard is now available at: 2024-01-13T08:17:23.605 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-01-13T08:17:23.605 INFO:teuthology.orchestra.run.smithi099.stderr: URL: https://smithi099:8443/ 2024-01-13T08:17:23.605 INFO:teuthology.orchestra.run.smithi099.stderr: User: admin 2024-01-13T08:17:23.605 INFO:teuthology.orchestra.run.smithi099.stderr: Password: gb6ek742cr 2024-01-13T08:17:23.605 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-01-13T08:17:23.605 INFO:teuthology.orchestra.run.smithi099.stderr:Enabling autotune for osd_memory_target 2024-01-13T08:17:23.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:23 smithi099 conmon[33313]: audit 2024-01-13T08:17:23.002763+0000 mon.smithi099 (mon.0) 91 : audit [DBG] from='client.? 172.21.15.99:0/3654793951' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-01-13T08:17:25.397 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-01-13T08:17:25.944 INFO:teuthology.orchestra.run.smithi099.stderr:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-01-13T08:17:25.944 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-01-13T08:17:25.944 INFO:teuthology.orchestra.run.smithi099.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-01-13T08:17:25.944 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-01-13T08:17:25.944 INFO:teuthology.orchestra.run.smithi099.stderr:Or, if you are only running a single cluster on this host: 2024-01-13T08:17:25.944 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-01-13T08:17:25.944 INFO:teuthology.orchestra.run.smithi099.stderr: sudo /home/ubuntu/cephtest/cephadm shell 2024-01-13T08:17:25.945 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-01-13T08:17:25.945 INFO:teuthology.orchestra.run.smithi099.stderr:Please consider enabling telemetry to help improve Ceph: 2024-01-13T08:17:25.945 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-01-13T08:17:25.945 INFO:teuthology.orchestra.run.smithi099.stderr: ceph telemetry on 2024-01-13T08:17:25.945 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-01-13T08:17:25.945 INFO:teuthology.orchestra.run.smithi099.stderr:For more information see: 2024-01-13T08:17:25.945 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-01-13T08:17:25.945 INFO:teuthology.orchestra.run.smithi099.stderr: https://docs.ceph.com/en/pacific/mgr/telemetry/ 2024-01-13T08:17:25.945 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-01-13T08:17:25.945 INFO:teuthology.orchestra.run.smithi099.stderr:Bootstrap complete. 2024-01-13T08:17:25.984 INFO:tasks.cephadm:Fetching config... 2024-01-13T08:17:25.984 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-01-13T08:17:25.984 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-01-13T08:17:26.002 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-01-13T08:17:26.002 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-01-13T08:17:26.002 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-01-13T08:17:26.061 INFO:tasks.cephadm:Fetching mon keyring... 2024-01-13T08:17:26.061 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-01-13T08:17:26.061 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi099/keyring of=/dev/stdout 2024-01-13T08:17:26.137 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:25 smithi099 conmon[33313]: cluster 2024-01-13T08:17:24.844266+0000 mon.smithi099 (mon.0) 92 : cluster [DBG] mgrmap e13: smithi099.oodult(active, since 6s) 2024-01-13T08:17:26.137 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:25 smithi099 conmon[33313]: 2024-01-13T08:17:26.137 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:25 smithi099 conmon[33313]: audit 2024-01-13T08:17:25.397328+0000 mon.smithi099 (mon.0) 93 : audit [INF] from='client.? 172.21.15.99:0/1162771524' entity='client.admin' 2024-01-13T08:17:26.141 INFO:tasks.cephadm:Fetching pub ssh key... 2024-01-13T08:17:26.142 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-01-13T08:17:26.142 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-01-13T08:17:26.199 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-01-13T08:17:26.200 DEBUG:teuthology.orchestra.run.smithi099:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcirLo7lt3Tk+l/5ZacdAUCgBHrPhbFNeMS+qImMwzkCLvQz7gNQ6KQqyOjoHoW0BQzk2DTN4LP3pXibU7egufaBwV3kbI2xAIR4NxHPe5JhXLarAN6A698mtLwzsaDV/82PTu5X3Z+k6xuSZW/Q5Rp55jhcl2wYBcAPPozvXQwtFLUQDlsmr/BjEUzZu46Tgw2DZrZqowEzXp3KTYTsrTj27YuR+OsVO1YdqMlIhCWzB6H9FdSW7w9oKZs/swqyv5MmYFWV0cGpK3j68J/P9M6iyxRcOtBtY6vt2Srql1zJ+S987y8o/yPeaGL9WV5S/KqcPSOO5Tzx8xt34kPJx7TF3jlMH8LjiB//ruZ2d2If96mazrfB+8oxD/5ncwYB00jgYmKRoYWQq6HkBx1q1YdN5NFa05PzKGO69/XePx4QNvmwUL/Wn1KVNNGlEpJTkZm3It+OSs91V90wYa4X8J2tIVSCWzPZc4odLPC2zGxeclYkYDhx9tRArUAJ6vgqU= ceph-f1ba87a2-b1eb-11ee-95ac-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-13T08:17:26.278 INFO:teuthology.orchestra.run.smithi099.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcirLo7lt3Tk+l/5ZacdAUCgBHrPhbFNeMS+qImMwzkCLvQz7gNQ6KQqyOjoHoW0BQzk2DTN4LP3pXibU7egufaBwV3kbI2xAIR4NxHPe5JhXLarAN6A698mtLwzsaDV/82PTu5X3Z+k6xuSZW/Q5Rp55jhcl2wYBcAPPozvXQwtFLUQDlsmr/BjEUzZu46Tgw2DZrZqowEzXp3KTYTsrTj27YuR+OsVO1YdqMlIhCWzB6H9FdSW7w9oKZs/swqyv5MmYFWV0cGpK3j68J/P9M6iyxRcOtBtY6vt2Srql1zJ+S987y8o/yPeaGL9WV5S/KqcPSOO5Tzx8xt34kPJx7TF3jlMH8LjiB//ruZ2d2If96mazrfB+8oxD/5ncwYB00jgYmKRoYWQq6HkBx1q1YdN5NFa05PzKGO69/XePx4QNvmwUL/Wn1KVNNGlEpJTkZm3It+OSs91V90wYa4X8J2tIVSCWzPZc4odLPC2zGxeclYkYDhx9tRArUAJ6vgqU= ceph-f1ba87a2-b1eb-11ee-95ac-87774f69a715 2024-01-13T08:17:26.290 DEBUG:teuthology.orchestra.run.smithi131:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcirLo7lt3Tk+l/5ZacdAUCgBHrPhbFNeMS+qImMwzkCLvQz7gNQ6KQqyOjoHoW0BQzk2DTN4LP3pXibU7egufaBwV3kbI2xAIR4NxHPe5JhXLarAN6A698mtLwzsaDV/82PTu5X3Z+k6xuSZW/Q5Rp55jhcl2wYBcAPPozvXQwtFLUQDlsmr/BjEUzZu46Tgw2DZrZqowEzXp3KTYTsrTj27YuR+OsVO1YdqMlIhCWzB6H9FdSW7w9oKZs/swqyv5MmYFWV0cGpK3j68J/P9M6iyxRcOtBtY6vt2Srql1zJ+S987y8o/yPeaGL9WV5S/KqcPSOO5Tzx8xt34kPJx7TF3jlMH8LjiB//ruZ2d2If96mazrfB+8oxD/5ncwYB00jgYmKRoYWQq6HkBx1q1YdN5NFa05PzKGO69/XePx4QNvmwUL/Wn1KVNNGlEpJTkZm3It+OSs91V90wYa4X8J2tIVSCWzPZc4odLPC2zGxeclYkYDhx9tRArUAJ6vgqU= ceph-f1ba87a2-b1eb-11ee-95ac-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-13T08:17:26.350 INFO:teuthology.orchestra.run.smithi131.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcirLo7lt3Tk+l/5ZacdAUCgBHrPhbFNeMS+qImMwzkCLvQz7gNQ6KQqyOjoHoW0BQzk2DTN4LP3pXibU7egufaBwV3kbI2xAIR4NxHPe5JhXLarAN6A698mtLwzsaDV/82PTu5X3Z+k6xuSZW/Q5Rp55jhcl2wYBcAPPozvXQwtFLUQDlsmr/BjEUzZu46Tgw2DZrZqowEzXp3KTYTsrTj27YuR+OsVO1YdqMlIhCWzB6H9FdSW7w9oKZs/swqyv5MmYFWV0cGpK3j68J/P9M6iyxRcOtBtY6vt2Srql1zJ+S987y8o/yPeaGL9WV5S/KqcPSOO5Tzx8xt34kPJx7TF3jlMH8LjiB//ruZ2d2If96mazrfB+8oxD/5ncwYB00jgYmKRoYWQq6HkBx1q1YdN5NFa05PzKGO69/XePx4QNvmwUL/Wn1KVNNGlEpJTkZm3It+OSs91V90wYa4X8J2tIVSCWzPZc4odLPC2zGxeclYkYDhx9tRArUAJ6vgqU= ceph-f1ba87a2-b1eb-11ee-95ac-87774f69a715 2024-01-13T08:17:26.363 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-01-13T08:17:29.095 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-01-13T08:17:29.096 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-01-13T08:17:29.775 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:29 smithi099 conmon[33313]: audit 2024-01-13T08:17:28.494123+0000 mon.smithi099 (mon.0) 94 : audit [INF] from='client.? 172.21.15.99:0/12979746' entity='client.admin' 2024-01-13T08:17:30.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:30 smithi099 conmon[33313]: audit 2024-01-13T08:17:29.893816+0000 mon.smithi099 (mon.0) 95 : audit [INF] from='mgr.14156 172.21.15.99:0/2715542913' entity='mgr.smithi099.oodult' 2024-01-13T08:17:30.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:30 smithi099 conmon[33313]: audit 2024-01-13T08:17:30.445681+0000 mon.smithi099 (mon.0) 96 : audit [INF] from='mgr.14156 172.21.15.99:0/2715542913' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config rm", "who": "osd/host:smithi099", "name": "osd_memory_target"}]: dispatch 2024-01-13T08:17:30.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:30 smithi099 conmon[33313]: audit 2024-01-13T08:17:30.448766+0000 mon.smithi099 (mon.0) 97 : audit [INF] from='mgr.14156 172.21.15.99:0/2715542913' entity='mgr.smithi099.oodult' 2024-01-13T08:17:30.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:30 smithi099 conmon[33313]: cephadm 2024-01-13T08:17:30.456869+0000 mgr.smithi099.oodult (mgr.14156) 5 : cephadm [INF] Deploying daemon alertmanager.smithi099 on smithi099 2024-01-13T08:17:31.649 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi131 2024-01-13T08:17:31.650 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-01-13T08:17:31.650 DEBUG:teuthology.orchestra.run.smithi131:> dd of=/etc/ceph/ceph.conf 2024-01-13T08:17:31.670 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-01-13T08:17:31.670 DEBUG:teuthology.orchestra.run.smithi131:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-13T08:17:31.730 INFO:tasks.cephadm:Adding host smithi131 to orchestrator... 2024-01-13T08:17:31.730 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph orch host add smithi131 2024-01-13T08:17:32.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:32 smithi099 conmon[33313]: audit 2024-01-13T08:17:31.076120+0000 mgr.smithi099.oodult (mgr.14156) 6 : audit 2024-01-13T08:17:32.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:32 smithi099 conmon[33313]: [DBG] from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T08:17:32.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:32 smithi099 conmon[33313]: audit 2024-01-13T08:17:31.081118+0000 mon.smithi099 (mon.0) 98 : audit [INF] from='mgr.14156 172.21.15.99:0/2715542913' entity='mgr.smithi099.oodult' 2024-01-13T08:17:38.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:38 smithi099 conmon[33313]: audit 2024-01-13T08:17:37.043317+0000 mon.smithi099 (mon.0) 99 : audit 2024-01-13T08:17:38.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:38 smithi099 conmon[33313]: [INF] from='mgr.14156 172.21.15.99:0/2715542913' entity='mgr.smithi099.oodult' 2024-01-13T08:17:38.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:38 smithi099 conmon[33313]: audit 2024-01-13T08:17:37.044866+0000 mon.smithi099 (mon.0) 100 : audit [INF] from='mgr.14156 172.21.15.99:0/2715542913' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi099", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-13T08:17:38.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:38 smithi099 conmon[33313]: audit 2024-01-13T08:17:37.046364+0000 mon.smithi099 (mon.0) 101 : audit [INF] from='mgr.14156 172.21.15.99:0/2715542913' entity='mgr.smithi099.oodult' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi099", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-01-13T08:17:38.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:38 smithi099 conmon[33313]: audit 2024-01-13T08:17:37.046891+0000 mon.smithi099 (mon.0) 102 : audit [DBG] from='mgr.14156 172.21.15.99:0/2715542913' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:17:38.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:38 smithi099 conmon[33313]: cephadm 2024-01-13T08:17:37.047324+0000 mgr.smithi099.oodult (mgr.14156) 7 : cephadm [INF] Deploying daemon crash.smithi099 on smithi099 2024-01-13T08:17:39.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:39 smithi099 conmon[33313]: cluster 2024-01-13T08:17:37.970922+0000 mgr.smithi099.oodult (mgr.14156 2024-01-13T08:17:39.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:39 smithi099 conmon[33313]: ) 8 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:17:39.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:39 smithi099 conmon[33313]: audit 2024-01-13T08:17:38.104855+0000 mon.smithi099 (mon.0) 103 : audit [INF] from='mgr.14156 172.21.15.99:0/2715542913' entity='mgr.smithi099.oodult' 2024-01-13T08:17:40.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:40 smithi099 conmon[33313]: audit 2024-01-13T08:17:39.166875+0000 mon.smithi099 (mon.0) 104 : audit [INF] from='mgr.14156 172.21.15.99:0/2715542913' entity='mgr.smithi099.oodult' 2024-01-13T08:17:40.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:40 smithi099 conmon[33313]: audit 2024-01-13T08:17:39.393859+0000 mon.smithi099 (mon.0) 105 : audit [INF] from='mgr.14156 172.21.15.99:0/2715542913' entity='mgr.smithi099.oodult' 2024-01-13T08:17:40.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:40 smithi099 conmon[33313]: audit 2024-01-13T08:17:39.399006+0000 mon.smithi099 (mon.0) 106 : audit [INF] from='mgr.14156 172.21.15.99:0/2715542913' entity='mgr.smithi099.oodult' 2024-01-13T08:17:40.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:40 smithi099 conmon[33313]: audit 2024-01-13T08:17:39.400967+0000 mon.smithi099 (mon.0) 107 : audit [INF] from='mgr.14156 172.21.15.99:0/2715542913' entity='mgr.smithi099.oodult' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-13T08:17:40.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:40 smithi099 conmon[33313]: audit 2024-01-13T08:17:39.401306+0000 mgr.smithi099.oodult (mgr.14156) 9 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-13T08:17:40.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:40 smithi099 conmon[33313]: audit 2024-01-13T08:17:39.406447+0000 mon.smithi099 (mon.0) 108 : audit [INF] from='mgr.14156 172.21.15.99:0/2715542913' entity='mgr.smithi099.oodult' 2024-01-13T08:17:40.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:40 smithi099 conmon[33313]: cephadm 2024-01-13T08:17:39.413444+0000 mgr.smithi099.oodult (mgr.14156) 10 : cephadm [INF] Deploying daemon grafana.smithi099 on smithi099 2024-01-13T08:17:41.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:41 smithi099 conmon[33313]: cluster 2024-01-13T08:17:39.971270+0000 mgr.smithi099.oodult (mgr.14156) 11 : 2024-01-13T08:17:41.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:41 smithi099 conmon[33313]: cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:17:43.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:43 smithi099 conmon[33313]: cluster 2024-01-13T08:17:41.971522+0000 mgr.smithi099.oodult (mgr.14156) 12 : cluster [DBG] 2024-01-13T08:17:43.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:43 smithi099 conmon[33313]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:17:43.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:43 smithi099 conmon[33313]: audit 2024-01-13T08:17:43.107705+0000 mon.smithi099 (mon.0) 109 : audit [INF] from='mgr.14156 172.21.15.99:0/2715542913' entity='mgr.smithi099.oodult' 2024-01-13T08:17:45.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:45 smithi099 conmon[33313]: cluster 2024-01-13T08:17:43.971770+0000 mgr.smithi099.oodult 2024-01-13T08:17:45.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:45 smithi099 conmon[33313]: (mgr.14156) 13 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:17:47.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:46 smithi099 conmon[33313]: cluster 2024-01-13T08:17:45.971993+0000 mgr.smithi099.oodult (mgr.14156) 2024-01-13T08:17:47.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:46 smithi099 conmon[33313]: 14 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:17:49.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:49 smithi099 conmon[33313]: cluster 2024-01-13T08:17:47.972192+0000 mgr.smithi099.oodult (mgr.14156) 15 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:17:51.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:51 smithi099 conmon[33313]: cluster 2024-01-13T08:17:49.972446+0000 mgr.smithi099.oodult (mgr.14156) 16 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:17:53.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:53 smithi099 conmon[33313]: cluster 2024-01-13T08:17:51.972817+0000 mgr.smithi099.oodult (mgr.14156) 17 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:17:55.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:55 smithi099 conmon[33313]: cluster 2024-01-13T08:17:53.973116+0000 mgr.smithi099.oodult (mgr.14156) 18 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:17:55.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:55 smithi099 conmon[33313]: 2024-01-13T08:17:57.225 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:56 smithi099 conmon[33313]: cluster 2024-01-13T08:17:55.973348 2024-01-13T08:17:57.226 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:56 smithi099 conmon[33313]: +0000 mgr.smithi099.oodult (mgr.14156) 19 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:17:58.543 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:58 smithi099 conmon[33313]: audit 2024-01-13T08:17:57.438696+0000 mon.smithi099 (mon.0) 110 : audit [INF] from='mgr.14156 172.21.15.99:0/2715542913' entity='mgr.smithi099.oodult' 2024-01-13T08:17:58.544 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:58 smithi099 conmon[33313]: cephadm 2024-01-13T08:17:57.440713+0000 mgr.smithi099.oodult (mgr.14156) 20 : cephadm [INF] Deploying daemon node-exporter.smithi099 on smithi099 2024-01-13T08:17:58.544 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:58 smithi099 conmon[33313]: audit 2024- 2024-01-13T08:17:58.544 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:58 smithi099 conmon[33313]: 01-13T08:17:58.114090 2024-01-13T08:17:58.544 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:58 smithi099 conmon[33313]: +0000 mon.smithi099 (mon.0) 111 : audit [INF] 2024-01-13T08:17:58.544 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:58 smithi099 conmon[33313]: from='mgr.14156 172.21.15.99:0/2715542913' entity='mgr.smithi099.oodult' 2024-01-13T08:17:59.651 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:59 smithi099 conmon[33313]: cluster 2024-01-13T08:17:57.973555+0000 2024-01-13T08:17:59.652 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:17:59 smithi099 conmon[33313]: mgr.smithi099.oodult (mgr.14156) 21 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:18:01.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:01 smithi099 conmon[33313]: cluster 2024-01-13T08:17:59.973796+0000 mgr.smithi099.oodult (mgr. 2024-01-13T08:18:01.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:01 smithi099 conmon[33313]: 14156) 22 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:18:01.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:01 smithi099 conmon[33313]: audit 2024-01-13T08:18:00.682677+0000 mon.smithi099 2024-01-13T08:18:01.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:01 smithi099 conmon[33313]: (mon.0) 112 : audit [INF] from='mgr.14156 172.21.15.99:0/2715542913' entity='mgr.smithi099.oodult' 2024-01-13T08:18:01.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:01 smithi099 conmon[33313]: audit 2024-01 2024-01-13T08:18:01.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:01 smithi099 conmon[33313]: -13T08:18:00.684757+0000 2024-01-13T08:18:01.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:01 smithi099 conmon[33313]: mon.smithi099 (mon.0) 113 : audit 2024-01-13T08:18:01.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:01 smithi099 conmon[33313]: [INF] from='mgr.14156 172.21.15.99:0/2715542913' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-01-13T08:18:02.820 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:02 smithi099 conmon[33313]: audit 2024-01-13T08:18:01.684437+0000 mon.smithi099 (mon.0) 2024-01-13T08:18:02.820 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:02 smithi099 conmon[33313]: 114 : audit [INF] from='mgr.14156 172.21.15.99:0/2715542913' entity='mgr.smithi099.oodult' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-01-13T08:18:02.820 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:02 smithi099 conmon[33313]: cluster 2024-01-13T08:18:01.684483+0000 mon.smithi099 (mon.0) 115 : cluster [DBG] mgrmap e14: smithi099.oodult(active, since 43s) 2024-01-13T08:18:07.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:07 smithi099 conmon[33313]: cluster 2024-01-13T08:18:07.091931+0000 mon.smithi099 (mon.0) 116 : cluster [INF] Active manager daemon smithi099.oodult restarted 2024-01-13T08:18:07.240 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:07 smithi099 conmon[33313]: cluster 2024-01-13T08:18:07.092201+0000 2024-01-13T08:18:07.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:07 smithi099 conmon[33313]: mon.smithi099 (mon.0) 117 : cluster [INF] Activating manager daemon smithi099.oodult 2024-01-13T08:18:07.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:07 smithi099 conmon[33313]: cluster 2024-01-13T08:18:07.097419+0000 mon.smithi099 (mon.0) 118 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-01-13T08:18:07.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:07 smithi099 conmon[33313]: cluster 2024-01-13T08:18:07.097520+0000 mon.smithi099 (mon.0) 119 : cluster [DBG] mgrmap e15: smithi099.oodult(active, starting, since 0.00542078s) 2024-01-13T08:18:07.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:07 smithi099 conmon[33313]: audit 2024-01-13T08:18:07.098261+0000 mon.smithi099 (mon.0) 120 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-01-13T08:18:07.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:07 smithi099 conmon[33313]: audit 2024-01-13T08:18:07.098975+0000 mon.smithi099 (mon.0) 121 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mgr metadata", "who": "smithi099.oodult", "id": "smithi099.oodult"}]: dispatch 2024-01-13T08:18:07.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:07 smithi099 conmon[33313]: audit 2024-01-13T 2024-01-13T08:18:07.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:07 smithi099 conmon[33313]: 08:18:07.099634+0000 mon.smithi099 (mon.0) 122 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-13T08:18:07.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:07 smithi099 conmon[33313]: audit 2024-01-13T08:18:07.099809+0000 mon.smithi099 (mon. 2024-01-13T08:18:07.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:07 smithi099 conmon[33313]: 0) 123 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-13T08:18:07.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:07 smithi099 conmon[33313]: audit 2024-01-13T08:18:07.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:07 smithi099 conmon[33313]: 2024-01-13T08:18:07.099971+0000 mon.smithi099 (mon. 2024-01-13T08:18:07.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:07 smithi099 conmon[33313]: 0) 124 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-13T08:18:07.241 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:07 smithi099 conmon[33313]: cluster 2024-01-13T08 2024-01-13T08:18:07.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:07 smithi099 conmon[33313]: :18:07.113901+0000 mon.smithi099 (mon.0) 2024-01-13T08:18:07.242 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:07 smithi099 conmon[33313]: 125 : cluster [INF] Manager daemon smithi099.oodult is now available 2024-01-13T08:18:08.320 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:08 smithi099 conmon[33313]: audit 2024-01-13T08:18:07.211300+0000 mon.smithi099 (mon 2024-01-13T08:18:08.320 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:08 smithi099 conmon[33313]: .0) 126 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:18:08.320 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:08 smithi099 conmon[33313]: audit 2024-01-13T08:18:07.212508+0000 mon.smithi099 (mon.0) 127 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:18:08.320 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:08 smithi099 conmon[33313]: audit 2024-01-13T08:18:07.237240+0000 mon.smithi099 (mon.0) 128 : audit [INF] 2024-01-13T08:18:08.320 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:08 smithi099 conmon[33313]: from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:08.320 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:08 smithi099 conmon[33313]: audit 2024-01-13T08:18:07 2024-01-13T08:18:08.320 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:08 smithi099 conmon[33313]: .248000+0000 mon.smithi099 (mon.0) 129 : audit [INF] 2024-01-13T08:18:08.320 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:08 smithi099 conmon[33313]: from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi099.oodult/mirror_snapshot_schedule"}]: dispatch 2024-01-13T08:18:08.321 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:08 smithi099 conmon[33313]: audit 2024 2024-01-13T08:18:08.321 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:08 smithi099 conmon[33313]: -01-13T08:18:07.253511+0000 mon.smithi099 (mon.0) 2024-01-13T08:18:08.321 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:08 smithi099 conmon[33313]: 130 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi099.oodult/trash_purge_schedule"}]: dispatch 2024-01-13T08:18:08.321 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:08 smithi099 conmon[33313]: cluster 2024-01-13T08:18: 2024-01-13T08:18:08.321 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:08 smithi099 conmon[33313]: 08.099009+0000 mon.smithi099 (mon.0) 131 : cluster 2024-01-13T08:18:08.321 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:08 smithi099 conmon[33313]: [DBG] mgrmap e16: smithi099.oodult(active, since 1.00691s) 2024-01-13T08:18:10.271 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:10 smithi099 conmon[33313]: cluster 2024-01-13T08:18:09.146983+0000 mon.smithi099 (mon.0 2024-01-13T08:18:10.272 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:10 smithi099 conmon[33313]: ) 132 : cluster [DBG] mgrmap e17: smithi099.oodult(active, since 2s) 2024-01-13T08:18:11.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:11 smithi099 conmon[33313]: audit 2024-01-13T08:18:10.241294+0000 mon.smithi099 (mon. 2024-01-13T08:18:11.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:11 smithi099 conmon[33313]: 0) 133 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:11.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:11 smithi099 conmon[33313]: audit 2024-01-13T08:18:10.548192+0000 mon.smithi099 (mon.0) 134 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config rm", "who": "osd/host:smithi099", "name": "osd_memory_target"}]: dispatch 2024-01-13T08:18:11.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:11 smithi099 conmon[33313]: 2024-01-13T08:18:11.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:11 smithi099 conmon[33313]: audit 2024- 2024-01-13T08:18:11.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:11 smithi099 conmon[33313]: 01-13T08:18:10.548943+0000 2024-01-13T08:18:11.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:11 smithi099 conmon[33313]: mon.smithi099 (mon.0) 135 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:18:11.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:11 smithi099 conmon[33313]: audit 2024 2024-01-13T08:18:11.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:11 smithi099 conmon[33313]: -01-13T08:18: 2024-01-13T08:18:11.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:11 smithi099 conmon[33313]: 10.549495+0000 mon.smithi099 (mon.0) 2024-01-13T08:18:11.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:11 smithi099 conmon[33313]: 136 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T08:18:11.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:11 smithi099 conmon[33313]: cephadm 2024-01- 2024-01-13T08:18:11.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:11 smithi099 conmon[33313]: 13T08:18:10.550524 2024-01-13T08:18:11.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:11 smithi099 conmon[33313]: +0000 mgr.smithi099.oodult (mgr.14180) 1 : cephadm [INF] 2024-01-13T08:18:11.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:11 smithi099 conmon[33313]: Updating smithi099:/etc/ceph/ceph.conf 2024-01-13T08:18:11.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:11 smithi099 conmon[33313]: cephadm 2024-01-13 2024-01-13T08:18:11.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:11 smithi099 conmon[33313]: T08:18:10.577644+0000 2024-01-13T08:18:11.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:11 smithi099 conmon[33313]: mgr.smithi099.oodult (mgr.14180) 2 : cephadm [INF] Updating smithi099:/etc/ceph/ceph.client.admin.keyring 2024-01-13T08:18:11.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:11 smithi099 conmon[33313]: 2024-01-13T08:18:11.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:11 smithi099 conmon[33313]: audit 2024-01-13T08:18:11.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:11 smithi099 conmon[33313]: 2024-01-13T08:18:10. 2024-01-13T08:18:11.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:11 smithi099 conmon[33313]: 614764+0000 mon.smithi099 (mon.0) 137 : audit [INF] 2024-01-13T08:18:11.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:11 smithi099 conmon[33313]: from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:11.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:11 smithi099 conmon[33313]: audit 2024-01-13 2024-01-13T08:18:11.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:11 smithi099 conmon[33313]: T08:18:10.617221+0000 2024-01-13T08:18:11.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:11 smithi099 conmon[33313]: mon.smithi099 (mon.0) 138 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:11.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:11 smithi099 conmon[33313]: 2024-01-13T08:18:11.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:11 smithi099 conmon[33313]: cephadm 2024-01-13T08:18:11.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:11 smithi099 conmon[33313]: 2024-01- 2024-01-13T08:18:11.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:11 smithi099 conmon[33313]: 13T08:18:10.628572+0000 mgr.smithi099.oodult 2024-01-13T08:18:11.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:11 smithi099 conmon[33313]: (mgr.14180) 3 : cephadm [INF] Deploying daemon prometheus.smithi099 on smithi099 2024-01-13T08:18:12.153 INFO:teuthology.orchestra.run.smithi131.stdout:Added host 'smithi131' with addr '172.21.15.131' 2024-01-13T08:18:12.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:12 smithi099 conmon[33313]: audit 2024-01-13T08:18:11.532583+0000 mgr.smithi099.oodult (mgr.14180) 4 2024-01-13T08:18:12.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:12 smithi099 conmon[33313]: : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi131", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T08:18:12.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:12 smithi099 conmon[33313]: audit 2024-01-13T08:18:12.155365+0000 mon.smithi099 (mon.0) 139 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:12.784 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph orch host ls --format=json 2024-01-13T08:18:13.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:13 smithi099 conmon[33313]: cephadm 2024-01-13T08:18:12.155735+0000 mgr.smithi099.oodult (mgr.14180) 5 : cephadm 2024-01-13T08:18:13.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:13 smithi099 conmon[33313]: [INF] Added host smithi131 2024-01-13T08:18:14.615 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-01-13T08:18:14.615 INFO:teuthology.orchestra.run.smithi131.stdout:[{"addr": "172.21.15.99", "hostname": "smithi099", "labels": [], "status": ""}, {"addr": "172.21.15.131", "hostname": "smithi131", "labels": [], "status": ""}] 2024-01-13T08:18:15.177 INFO:tasks.cephadm:Setting crush tunables to default 2024-01-13T08:18:15.177 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd crush tunables default 2024-01-13T08:18:15.204 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:14 smithi099 conmon[33313]: audit 2024-01-13T08:18:14.618235+0000 mgr.smithi099.oodult (mgr.14180) 6 2024-01-13T08:18:15.205 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:14 smithi099 conmon[33313]: : audit [DBG] from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-13T08:18:20.783 INFO:teuthology.orchestra.run.smithi099.stderr:adjusted tunables profile to default 2024-01-13T08:18:20.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:20 smithi099 conmon[33313]: audit 2024-01-13T08:18:20.730545+0000 mon.smithi099 (mon.0) 140 : audit [INF] from='client.? 172.21.15.99:0/2819508267' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-01-13T08:18:21.716 INFO:tasks.cephadm:Adding mon.smithi099 on smithi099 2024-01-13T08:18:21.716 INFO:tasks.cephadm:Adding mon.smithi131 on smithi131 2024-01-13T08:18:21.717 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph orch apply mon '2;smithi099:172.21.15.99=smithi099;smithi131:172.21.15.131=smithi131' 2024-01-13T08:18:21.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:21 smithi099 conmon[33313]: audit 2024-01-13T08:18:20.783089+0000 2024-01-13T08:18:21.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:21 smithi099 conmon[33313]: mon.smithi099 (mon.0) 141 : audit [INF] from='client.? 172.21.15.99:0/2819508267' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-01-13T08:18:21.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:21 smithi099 conmon[33313]: cluster 2024-01-13T08:18:20.783150+0000 mon.smithi099 (mon.0) 142 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-01-13T08:18:23.609 INFO:teuthology.orchestra.run.smithi131.stdout:Scheduled mon update... 2024-01-13T08:18:23.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:23 smithi099 conmon[33313]: audit 2024-01-13T08:18:22.286008+0000 mon.smithi099 (mon.0) 143 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:23.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:23 smithi099 conmon[33313]: audit 2024-01-13T08:18:22.287104+0000 mon.smithi099 (mon.0) 144 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:18:23.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:23 smithi099 conmon[33313]: audit 2024-01-13T08:18:22.825321+0000 mon.smithi099 (mon.0) 145 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:24.419 DEBUG:teuthology.orchestra.run.smithi131:mon.smithi131> sudo journalctl -f -n 0 -u ceph-f1ba87a2-b1eb-11ee-95ac-87774f69a715@mon.smithi131.service 2024-01-13T08:18:24.421 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T08:18:24.421 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T08:18:24.451 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:-- Logs begin at Sat 2024-01-13 08:04:55 UTC. -- 2024-01-13T08:18:24.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:24 smithi099 conmon[33313]: audit 2024-01-13T08:18:23.359607+0000 mon.smithi099 (mon.0) 146 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:24.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:24 smithi099 conmon[33313]: audit 2024-01-13T08:18:23.607936+0000 mgr.smithi099.oodult (mgr.14180) 7 : audit [DBG] from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi099:172.21.15.99=smithi099;smithi131:172.21.15.131=smithi131", "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T08:18:24.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:24 smithi099 conmon[33313]: cephadm 2024-01-13T08:18:23.610074+0000 mgr.smithi099.oodult (mgr.14180) 8 : cephadm [INF] Saving service mon spec with placement smithi099:172.21.15.99=smithi099;smithi131:172.21.15.131=smithi131;count:2 2024-01-13T08:18:24.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:24 smithi099 conmon[33313]: audit 2024-01-13T08:18:23.612433+0000 mon.smithi099 (mon.0) 147 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:26.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:25 smithi099 conmon[33313]: audit 2024-01-13T08:18:24.928008+0000 mon.smithi099 (mon.0) 148 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:26.980 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-01-13T08:18:26.980 INFO:teuthology.orchestra.run.smithi131.stdout:{"epoch":1,"fsid":"f1ba87a2-b1eb-11ee-95ac-87774f69a715","modified":"2024-01-13T08:16:29.216397Z","created":"2024-01-13T08:16:29.216397Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi099","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:3300","nonce":0},{"type":"v1","addr":"172.21.15.99:6789","nonce":0}]},"addr":"172.21.15.99:6789/0","public_addr":"172.21.15.99:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T08:18:26.982 INFO:teuthology.orchestra.run.smithi131.stderr:dumped monmap epoch 1 2024-01-13T08:18:27.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:27 smithi099 conmon[33313]: audit 2024-01-13T08:18:26.983437+0000 mon.smithi099 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.131:0/278418538' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T08:18:28.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:28 smithi099 conmon[33313]: cluster 2024-01-13T08:18:27.101129+0000 mgr.smithi099.oodult (mgr.14180) 9 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:18:28.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:28 smithi099 conmon[33313]: audit 2024-01-13T08:18:27.239832+0000 mon.smithi099 (mon.0) 150 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:28.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:28 smithi099 conmon[33313]: audit 2024-01-13T08:18:28.063702+0000 mon.smithi099 (mon.0) 151 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:28.847 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T08:18:28.847 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T08:18:29.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:29 smithi099 conmon[33313]: audit 2024-01-13T08:18:28.414408+0000 mon.smithi099 (mon.0) 152 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:29.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:29 smithi099 conmon[33313]: audit 2024-01-13T08:18:28.415271+0000 mon.smithi099 (mon. 2024-01-13T08:18:29.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:29 smithi099 conmon[33313]: 0) 153 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config rm", "who": "osd/host:smithi131", "name": "osd_memory_target"}]: dispatch 2024-01-13T08:18:29.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:29 smithi099 conmon[33313]: audit 2024-01-13T08:18:28.416182+0000 mon.smithi099 (mon.0) 154 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:18:29.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:29 smithi099 conmon[33313]: audit 2024-01-13T08:18:28.416857+0000 mon.smithi099 (mon.0) 155 2024-01-13T08:18:29.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:29 smithi099 conmon[33313]: : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T08:18:29.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:29 smithi099 conmon[33313]: cephadm 2024-01-13T08:18 2024-01-13T08:18:29.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:29 smithi099 conmon[33313]: :28.418015+0000 mgr.smithi099.oodult (mgr.14180 2024-01-13T08:18:29.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:29 smithi099 conmon[33313]: ) 10 : cephadm [INF] Updating smithi131:/etc/ceph/ceph.conf 2024-01-13T08:18:29.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:29 smithi099 conmon[33313]: cephadm 2024-01-13 2024-01-13T08:18:29.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:29 smithi099 conmon[33313]: T08:18:28.435684+0000 mgr.smithi099.oodult (mgr.14180) 11 : cephadm [INF] Updating smithi131:/etc/ceph/ceph.client.admin.keyring 2024-01-13T08:18:29.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:29 smithi099 conmon[33313]: audit 2024-01-13T08 2024-01-13T08:18:29.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:29 smithi099 conmon[33313]: :18:28.463175+0000 mon.smithi099 (mon.0) 156 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:29.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:29 smithi099 conmon[33313]: audit 2024-01-13 2024-01-13T08:18:29.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:29 smithi099 conmon[33313]: T08:18:28.468871+0000 mon.smithi099 (mon.0) 157 : audit [INF] 2024-01-13T08:18:29.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:29 smithi099 conmon[33313]: from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:29.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:29 smithi099 conmon[33313]: audit 2024-01-13T08:18:28.470598 2024-01-13T08:18:29.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:29 smithi099 conmon[33313]: +0000 mon.smithi099 (mon.0) 158 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi131", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-13T08:18:29.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:29 smithi099 conmon[33313]: audit 2024-01-13T08:18:29.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:29 smithi099 conmon[33313]: 2024-01-13T08:18:28.472674+0000 mon.smithi099 (mon.0 2024-01-13T08:18:29.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:29 smithi099 conmon[33313]: ) 159 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi131", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-01-13T08:18:29.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:29 smithi099 conmon[33313]: audit 2024-01- 2024-01-13T08:18:29.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:29 smithi099 conmon[33313]: 13T08:18:28.473513+0000 mon.smithi099 (mon.0) 160 : audit 2024-01-13T08:18:29.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:29 smithi099 conmon[33313]: [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:18:29.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:29 smithi099 conmon[33313]: cephadm 2024-01-13 2024-01-13T08:18:29.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:29 smithi099 conmon[33313]: T08:18:28.474226+0000 mgr.smithi099.oodult (mgr.14180) 12 : cephadm [INF] 2024-01-13T08:18:29.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:29 smithi099 conmon[33313]: Deploying daemon crash.smithi131 on smithi131 2024-01-13T08:18:30.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:30 smithi099 conmon[33313]: cluster 2024-01-13T08:18:29.101520+0000 mgr.smithi099.oodult (mgr.14180) 13 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:18:31.181 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-01-13T08:18:31.181 INFO:teuthology.orchestra.run.smithi131.stdout:{"epoch":1,"fsid":"f1ba87a2-b1eb-11ee-95ac-87774f69a715","modified":"2024-01-13T08:16:29.216397Z","created":"2024-01-13T08:16:29.216397Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi099","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:3300","nonce":0},{"type":"v1","addr":"172.21.15.99:6789","nonce":0}]},"addr":"172.21.15.99:6789/0","public_addr":"172.21.15.99:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T08:18:31.183 INFO:teuthology.orchestra.run.smithi131.stderr:dumped monmap epoch 1 2024-01-13T08:18:32.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:31 smithi099 conmon[33313]: cluster 2024-01-13T08:18:31.101916+0000 mgr.smithi099.oodult (mgr.14180) 14 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:18:32.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:31 smithi099 conmon[33313]: 2024-01-13T08:18:32.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:31 smithi099 conmon[33313]: audit 2024-01-13T08:18:31.184021+0000 mon.smithi099 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.131:0/2647592232' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T08:18:32.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:31 smithi099 conmon[33313]: audit 2024-01-13T08:18:31.211843+0000 mon.smithi099 (mon.0) 162 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:32.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:31 smithi099 conmon[33313]: audit 2024-01-13T08:18:31.214103+0000 mon.smithi099 (mon.0) 163 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi131.qnazik", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-13T08:18:32.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:31 smithi099 conmon[33313]: audit 2024-01-13T08:18:31.215917+0000 mon.smithi099 (mon.0) 164 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi131.qnazik", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-01-13T08:18:32.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:31 smithi099 conmon[33313]: audit 2024-01-13T08:18:31.216604+0000 mon.smithi099 (mon.0) 165 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-13T08:18:32.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:31 smithi099 conmon[33313]: audit 2024-01-13T08:18:31.217326+0000 mon.smithi099 (mon.0) 166 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:18:32.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:31 smithi099 conmon[33313]: cephadm 2024-01-13T08:18:31.217853+0000 mgr.smithi099.oodult (mgr.14180) 15 : cephadm [INF] Deploying daemon mgr.smithi131.qnazik on smithi131 2024-01-13T08:18:32.829 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T08:18:32.829 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T08:18:33.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:33 smithi099 conmon[33313]: audit 2024-01-13T08:18:32.243072+0000 mon.smithi099 (mon.0) 167 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:34.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:34 smithi099 conmon[33313]: cluster 2024-01-13T08:18:33.102347+0000 mgr.smithi099.oodult (mgr.14180) 16 2024-01-13T08:18:34.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:34 smithi099 conmon[33313]: : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:18:34.982 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-01-13T08:18:34.983 INFO:teuthology.orchestra.run.smithi131.stdout:{"epoch":1,"fsid":"f1ba87a2-b1eb-11ee-95ac-87774f69a715","modified":"2024-01-13T08:16:29.216397Z","created":"2024-01-13T08:16:29.216397Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi099","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:3300","nonce":0},{"type":"v1","addr":"172.21.15.99:6789","nonce":0}]},"addr":"172.21.15.99:6789/0","public_addr":"172.21.15.99:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-13T08:18:34.985 INFO:teuthology.orchestra.run.smithi131.stderr:dumped monmap epoch 1 2024-01-13T08:18:35.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:35 smithi099 conmon[33313]: audit 2024-01-13T08:18:34.384761+0000 mon.smithi099 (mon.0) 168 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:35.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:35 smithi099 conmon[33313]: audit 2024-01-13T08:18:34.386844+0000 mon.smithi099 (mon.0) 169 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-13T08:18:35.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:35 smithi099 conmon[33313]: audit 2024-01-13T08:18:34.387675+0000 mon.smithi099 (mon.0) 170 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:18:35.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:35 smithi099 conmon[33313]: cephadm 2024 2024-01-13T08:18:35.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:35 smithi099 conmon[33313]: -01-13T08:18:34.388330+0000 mgr.smithi099.oodult (mgr.14180) 17 : cephadm [INF] Deploying daemon mon.smithi131 on smithi131 2024-01-13T08:18:35.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:35 smithi099 conmon[33313]: audit 2024-01-13T08:18:34.985979+0000 mon.smithi099 (mon.0 2024-01-13T08:18:35.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:35 smithi099 conmon[33313]: ) 171 : audit [DBG] from='client.? 172.21.15.131:0/3540337368' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T08:18:36.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:36 smithi099 conmon[33313]: cluster 2024-01-13T08:18:35.102565+0000 mgr.smithi099.oodult (mgr.14180) 18 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:18:36.934 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-13T08:18:36.934 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph mon dump -f json 2024-01-13T08:18:38.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:38 smithi099 conmon[33313]: cluster 2024-01-13T08:18:37.102894+0000 mgr.smithi099.oodult (mgr.14180) 19 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:18:38.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:38 smithi099 conmon[33313]: audit 2024-01-13T08:18:37.246600+0000 mon.smithi099 (mon.0) 172 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:38.788 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:38 smithi131 conmon[38024]: debug 2024-01-13T08:18:38.564+0000 7f4a16b0e700 1 mon.smithi131@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-01-13T08:18:43.950 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: audit 2024-01-13T08:18:38.572990+0000 mon.smithi099 (mon.0) 174 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-01-13T08:18:43.950 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: audit 2024-01-13T08:18:38.573184+0000 mon.smithi099 (mon.0) 175 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2024-01-13T08:18:43.950 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: cluster 2024-01-13T08:18:38.573303+0000 mon.smithi099 (mon.0) 176 : cluster [INF] mon.smithi099 calling monitor election 2024-01-13T08:18:43.951 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: cluster 2024-01-13T08:18:39.103264+0000 mgr.smithi099.oodult (mgr.14180) 20 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:18:43.951 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: 2024-01-13T08:18:43.951 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: audit 2024-01-13T08:18:39.569390+0000 mon.smithi099 (mon.0) 177 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2024-01-13T08:18:43.951 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: audit 2024 2024-01-13T08:18:43.951 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: -01-13T08:18:39.857456+0000 mon.smithi099 (mon.0) 178 : audit [DBG] from='mgr.? 172.21.15.131:0/2807353175' entity='mgr.smithi131.qnazik' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi131.qnazik/crt"}]: dispatch 2024-01-13T08:18:43.951 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: audit 2024-01-13T08 2024-01-13T08:18:43.951 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: :18:40.569574+0000 mon.smithi099 (mon.0) 179 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2024-01-13T08:18:43.951 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: cluster 2024-01 2024-01-13T08:18:43.951 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: -13T08:18:40.570949+0000 mon.smithi131 (mon.1) 1 : cluster [INF] mon.smithi131 calling monitor election 2024-01-13T08:18:43.951 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: cluster 2024-01-13T08:18:41.103592+0000 mgr.smithi099.oodult (mgr.14180) 21 : cluster 2024-01-13T08:18:43.951 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:18:43.951 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: audit 2024-01-13T08: 2024-01-13T08:18:43.951 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: 18:41.570067+0000 mon.smithi099 (mon.0) 180 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2024-01-13T08:18:43.951 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: audit 2024-01-13T08:18:42.570093+0000 mon.smithi099 (mon. 2024-01-13T08:18:43.952 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: 0) 181 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2024-01-13T08:18:43.952 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: audit 2024-01-13T08:18:43.952 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: 2024-01-13T08:18:43.569986+0000 mon.smithi099 (mon.0) 182 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2024-01-13T08:18:43.952 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: cluster 2024-01-13T08:18:43.600247 2024-01-13T08:18:43.952 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: +0000 mon.smithi099 (mon.0) 183 : cluster [INF] mon.smithi099 is new leader, mons smithi099,smithi131 in quorum (ranks 0,1) 2024-01-13T08:18:43.952 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: cluster 2024-01-13T08:18:43.606202+0000 mon.smithi099 (mon.0 2024-01-13T08:18:43.952 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: ) 184 : cluster [DBG] monmap e2: 2 mons at {smithi099=[v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0],smithi131=[v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0]} removed_ranks: {} 2024-01-13T08:18:43.952 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: cluster 2024 2024-01-13T08:18:43.952 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: -01-13T08:18:43.618588+0000 mon.smithi099 (mon.0) 185 : cluster [DBG] fsmap 2024-01-13T08:18:43.952 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: cluster 2024-01-13T08:18:43.618615+0000 2024-01-13T08:18:43.952 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: mon.smithi099 (mon.0) 186 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-01-13T08:18:43.952 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: cluster 2024-01-13T08:18:43.618841+0000 mon.smithi099 (mon.0) 187 2024-01-13T08:18:43.952 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: : cluster [DBG] mgrmap e17: smithi099.oodult(active, since 36s) 2024-01-13T08:18:43.952 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: cluster 2024-01-13T 2024-01-13T08:18:43.953 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: 08:18:43.619811+0000 mon.smithi099 (mon.0) 188 : cluster [DBG] Standby manager daemon smithi131.qnazik started 2024-01-13T08:18:43.953 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: audit 2024-01-13T08:18:43.953 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: 2024-01-13T08:18:43.620950+0000 mon.smithi099 (mon.0) 189 : audit [DBG] from='mgr.? 172.21.15.131:0/2807353175' entity='mgr.smithi131.qnazik' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-01-13T08:18:43.953 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: cluster 2024-01-13T08:18:43.622366+0000 mon.smithi099 (mon 2024-01-13T08:18:43.953 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: .0) 190 : cluster [INF] overall HEALTH_OK 2024-01-13T08:18:43.953 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: audit 2024-01-13T08:18:43.953 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: 2024-01-13T08:18:43.622812+0000 mon.smithi099 (mon.0) 191 : audit [DBG] from='mgr.? 172.21.15.131:0/2807353175' entity='mgr.smithi131.qnazik' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi131.qnazik/key"}]: dispatch 2024-01-13T08:18:43.953 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: audit 2024-01-13T08:18:43.623327+0000 2024-01-13T08:18:43.953 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: mon.smithi099 (mon.0) 192 : audit [DBG] from='mgr.? 172.21.15.131:0/2807353175' entity='mgr.smithi131.qnazik' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-01-13T08:18:43.953 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: audit 2024-01-13T08:18:43.627304+0000 mon.smithi099 (mon.0 2024-01-13T08:18:43.953 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:43 smithi131 conmon[38024]: ) 193 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:43.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: audit 2024-01-13T08:18:38.572990+0000 mon.smithi099 (mon.0) 174 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mon metadata", "id": "smithi099"}]: dispatch 2024-01-13T08:18:43.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: audit 2024-01-13T08:18:38.573184+0000 mon.smithi099 (mon.0) 175 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2024-01-13T08:18:43.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: cluster 2024-01-13T08:18:38.573303+0000 mon.smithi099 (mon.0 2024-01-13T08:18:43.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: ) 176 : cluster [INF] mon.smithi099 calling monitor election 2024-01-13T08:18:43.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: cluster 2024-01-13T08 2024-01-13T08:18:43.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: :18:39.103264+0000 mgr.smithi099.oodult (mgr.14180) 20 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:18:43.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: audit 2024-01-13 2024-01-13T08:18:43.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: T08:18:39.569390+0000 mon.smithi099 (mon.0) 177 : 2024-01-13T08:18:43.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2024-01-13T08:18:43.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: audit 2024-01-13T08:18:39.857456 2024-01-13T08:18:43.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: +0000 mon.smithi099 (mon.0) 178 : audit [DBG] from='mgr.? 172.21.15.131:0/2807353175' entity='mgr.smithi131.qnazik' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi131.qnazik/crt"}]: dispatch 2024-01-13T08:18:43.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: audit 2024- 2024-01-13T08:18:43.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: 01-13T08:18:40.569574+0000 mon.smithi099 (mon.0) 2024-01-13T08:18:43.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: 179 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2024-01-13T08:18:43.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: cluster 2024-01-13T08:18 2024-01-13T08:18:43.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: :40.570949+0000 mon.smithi131 (mon.1) 1 : cluster [INF] 2024-01-13T08:18:43.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: mon.smithi131 calling monitor election 2024-01-13T08:18:43.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: cluster 2024-01-13T08:18:41.103592+0000 2024-01-13T08:18:43.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: mgr.smithi099.oodult (mgr.14180) 21 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:18:43.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: audit 2024- 2024-01-13T08:18:43.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: 01-13T08:18:41.570067+0000 mon.smithi099 (mon.0 2024-01-13T08:18:43.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: ) 180 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2024-01-13T08:18:43.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: audit 2024-01-13T 2024-01-13T08:18:43.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: 08:18:42.570093+0000 mon.smithi099 (mon.0) 181 2024-01-13T08:18:43.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2024-01-13T08:18:43.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: audit 2024-01-13T08:18: 2024-01-13T08:18:43.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: 43.569986+0000 mon.smithi099 (mon.0) 182 : audit [DBG] 2024-01-13T08:18:43.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2024-01-13T08:18:43.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: cluster 2024-01-13T08:18:43.600247+0000 2024-01-13T08:18:43.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: mon.smithi099 (mon.0) 183 : cluster [INF] mon.smithi099 is new leader, mons smithi099,smithi131 in quorum (ranks 0,1) 2024-01-13T08:18:43.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: cluster 2024-01-13T08:18:43.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: 2024-01-13T08:18:43.606202+0000 mon.smithi099 (mon.0 2024-01-13T08:18:43.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: ) 184 : cluster [DBG] monmap e2: 2 mons at {smithi099=[v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0],smithi131=[v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0]} removed_ranks: {} 2024-01-13T08:18:43.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: cluster 2024-01-13T08: 2024-01-13T08:18:43.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: 18:43.618588+0000 mon.smithi099 (mon.0) 185 : 2024-01-13T08:18:43.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: cluster [DBG] fsmap 2024-01-13T08:18:43.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: cluster 2024-01-13T08:18:43. 2024-01-13T08:18:43.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: 618615+0000 mon.smithi099 (mon.0) 186 : cluster [DBG] 2024-01-13T08:18:43.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: osdmap e5: 0 total, 0 up, 0 in 2024-01-13T08:18:43.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: cluster 2024-01-13T08:18:43.618841+0000 2024-01-13T08:18:43.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: mon.smithi099 (mon.0) 187 : cluster [DBG] mgrmap e17: smithi099.oodult(active, since 36s) 2024-01-13T08:18:43.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: cluster 2024 2024-01-13T08:18:43.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: -01-13T08:18:43.619811+0000 mon.smithi099 (mon. 2024-01-13T08:18:43.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: 0) 188 : cluster [DBG] Standby manager daemon smithi131.qnazik started 2024-01-13T08:18:43.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: audit 2024-01- 2024-01-13T08:18:43.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: 13T08:18:43.620950+0000 mon.smithi099 (mon.0) 2024-01-13T08:18:43.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: 189 : audit [DBG] from='mgr.? 172.21.15.131:0/2807353175' entity='mgr.smithi131.qnazik' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-01-13T08:18:43.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: cluster 2024-01-13 2024-01-13T08:18:43.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: T08:18:43.622366+0000 mon.smithi099 (mon.0) 190 2024-01-13T08:18:43.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: : cluster [INF] overall HEALTH_OK 2024-01-13T08:18:43.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: audit 2024-01-13T08:18:43.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: 2024-01-13T08:18:43.622812+0000 mon.smithi099 (mon.0) 191 : audit 2024-01-13T08:18:43.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: [DBG] from='mgr.? 172.21.15.131:0/2807353175' entity='mgr.smithi131.qnazik' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi131.qnazik/key"}]: dispatch 2024-01-13T08:18:43.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: audit 2024-01-13T08:18:43. 2024-01-13T08:18:43.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: 623327+0000 mon.smithi099 (mon.0) 192 : audit [DBG] 2024-01-13T08:18:43.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: from='mgr.? 172.21.15.131:0/2807353175' entity='mgr.smithi131.qnazik' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-01-13T08:18:43.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: audit 2024-01-13T08:18:43.627304+0000 mon.smithi099 2024-01-13T08:18:43.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:43 smithi099 conmon[33313]: (mon.0) 193 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:44.939 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:44 smithi131 conmon[38024]: cluster 2024-01-13T08:18:43.103960+0000 mgr.smithi099.oodult (mgr.14180) 22 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:18:44.940 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:44 smithi131 conmon[38024]: cephadm 2024-01-13T08:18:43.629394+0000 mgr.smithi099.oodult (mgr.14180) 23 : cephadm [INF] Deploying daemon node-exporter.smithi131 on smithi131 2024-01-13T08:18:44.940 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:44 smithi131 conmon[38024]: cluster 2024-01-13T08:18:43.679080+0000 mon.smithi099 (mon.0) 194 : cluster [DBG] mgrmap e18: smithi099.oodult(active, since 36s), standbys: smithi131.qnazik 2024-01-13T08:18:44.940 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:44 smithi131 conmon[38024]: audit 2024-01-13T08:18:43.679255+0000 mon.smithi099 (mon.0) 195 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mgr metadata", "who": "smithi131.qnazik", "id": "smithi131.qnazik"}]: dispatch 2024-01-13T08:18:44.940 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:44 smithi131 conmon[38024]: audit 2024-01-13T08:18:44.570398+0000 mon.smithi099 (mon.0) 196 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2024-01-13T08:18:44.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:44 smithi099 conmon[33313]: cluster 2024-01-13T08:18:43.103960+0000 mgr.smithi099.oodult (mgr.14180) 22 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:18:44.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:44 smithi099 conmon[33313]: cephadm 2024-01-13T08:18:43.629394+0000 mgr.smithi099.oodult (mgr.14180) 23 : cephadm [INF] Deploying daemon node-exporter.smithi131 on smithi131 2024-01-13T08:18:44.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:44 smithi099 conmon[33313]: cluster 2024-01-13T08:18:44.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:44 smithi099 conmon[33313]: 2024-01-13T08:18:43.679080+0000 mon.smithi099 (mon.0) 194 : cluster [DBG] 2024-01-13T08:18:44.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:44 smithi099 conmon[33313]: mgrmap e18: smithi099.oodult(active, since 36s), standbys: smithi131.qnazik 2024-01-13T08:18:44.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:44 smithi099 conmon[33313]: audit 2024-01-13T08:18:43.679255+0000 2024-01-13T08:18:44.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:44 smithi099 conmon[33313]: mon.smithi099 (mon.0) 195 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mgr metadata", "who": "smithi131.qnazik", "id": "smithi131.qnazik"}]: dispatch 2024-01-13T08:18:44.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:44 smithi099 conmon[33313]: audit 2024-01 2024-01-13T08:18:44.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:44 smithi099 conmon[33313]: -13T08:18:44.570398+0000 mon.smithi099 (mon.0) 196 2024-01-13T08:18:44.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:44 smithi099 conmon[33313]: : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mon metadata", "id": "smithi131"}]: dispatch 2024-01-13T08:18:45.105 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-01-13T08:18:45.105 INFO:teuthology.orchestra.run.smithi131.stdout:{"epoch":2,"fsid":"f1ba87a2-b1eb-11ee-95ac-87774f69a715","modified":"2024-01-13T08:18:38.569677Z","created":"2024-01-13T08:16:29.216397Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi099","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:3300","nonce":0},{"type":"v1","addr":"172.21.15.99:6789","nonce":0}]},"addr":"172.21.15.99:6789/0","public_addr":"172.21.15.99:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi131","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-01-13T08:18:45.106 INFO:teuthology.orchestra.run.smithi131.stderr:dumped monmap epoch 2 2024-01-13T08:18:45.892 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-01-13T08:18:45.892 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph config generate-minimal-conf 2024-01-13T08:18:45.928 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:45 smithi099 conmon[33313]: audit 2024-01-13T08:18:45.106195+0000 mon.smithi099 (mon.0) 197 : audit [DBG] from='client.? 172.21.15.131:0/2912493439' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T08:18:46.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:45 smithi131 conmon[38024]: audit 2024-01-13T08:18:45.106195+0000 mon.smithi099 (mon.0 2024-01-13T08:18:46.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:45 smithi131 conmon[38024]: ) 197 : audit [DBG] from='client.? 172.21.15.131:0/2912493439' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-13T08:18:46.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:46 smithi099 conmon[33313]: cluster 2024-01-13T08:18:45.104295+0000 mgr.smithi099.oodult (mgr.14180) 24 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:18:47.040 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:46 smithi131 conmon[38024]: cluster 2024-01-13T08:18:45.104295+0000 mgr.smithi099.oodult (mgr.14180) 2024-01-13T08:18:47.040 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:46 smithi131 conmon[38024]: 24 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:18:47.797 INFO:teuthology.orchestra.run.smithi099.stdout:# minimal ceph.conf for f1ba87a2-b1eb-11ee-95ac-87774f69a715 2024-01-13T08:18:47.797 INFO:teuthology.orchestra.run.smithi099.stdout:[global] 2024-01-13T08:18:47.797 INFO:teuthology.orchestra.run.smithi099.stdout: fsid = f1ba87a2-b1eb-11ee-95ac-87774f69a715 2024-01-13T08:18:47.797 INFO:teuthology.orchestra.run.smithi099.stdout: mon_host = [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] 2024-01-13T08:18:48.390 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-01-13T08:18:48.390 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-01-13T08:18:48.390 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/ceph/ceph.conf 2024-01-13T08:18:48.462 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-01-13T08:18:48.462 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-13T08:18:48.527 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:48 smithi099 conmon[33313]: cluster 2024-01-13T08:18:47.104523+0000 mgr.smithi099.oodult (mgr. 2024-01-13T08:18:48.527 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:48 smithi099 conmon[33313]: 14180) 25 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:18:48.527 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:48 smithi099 conmon[33313]: audit 2024-01-13T08:18:47.251470+0000 mon.smithi099 (mon.0) 198 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:48.527 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:48 smithi099 conmon[33313]: audit 2024-01-13T08:18:47.266134+0000 mon.smithi099 (mon.0) 199 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:48.527 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:48 smithi099 conmon[33313]: audit 2024-01-13T08:18:47.267697+0000 mon.smithi099 (mon.0) 200 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:18:48.527 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:48 smithi099 conmon[33313]: audit 2024-01-13T08:18:47.800292+0000 mon.smithi099 (mon.0) 201 : audit [DBG] from='client.? 172.21.15.99:0/982613696' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:18:48.530 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-01-13T08:18:48.530 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/ceph/ceph.conf 2024-01-13T08:18:48.558 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:48 smithi131 conmon[38024]: cluster 2024-01-13T08:18:47.104523+0000 mgr.smithi099.oodult (mgr.14180) 25 : cluster 2024-01-13T08:18:48.558 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:48 smithi131 conmon[38024]: [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:18:48.558 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:48 smithi131 conmon[38024]: audit 2024-01-13T08:18:47.251470+0000 mon.smithi099 (mon.0) 198 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:48.558 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:48 smithi131 conmon[38024]: audit 2024-01-13T08:18:47.266134+0000 mon.smithi099 (mon.0) 199 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:48.558 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:48 smithi131 conmon[38024]: audit 2024-01-13T08:18:47.267697+0000 mon.smithi099 (mon.0) 200 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:18:48.558 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:48 smithi131 conmon[38024]: audit 2024-01-13T08:18:47.800292+0000 mon.smithi099 (mon 2024-01-13T08:18:48.558 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:48 smithi131 conmon[38024]: .0) 201 : audit [DBG] from='client.? 172.21.15.99:0/982613696' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:18:48.562 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-01-13T08:18:48.562 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-13T08:18:48.630 INFO:tasks.cephadm:Deploying OSDs... 2024-01-13T08:18:48.630 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-01-13T08:18:48.630 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/scratch_devs of=/dev/stdout 2024-01-13T08:18:48.648 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-01-13T08:18:48.648 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/nvme1n1 2024-01-13T08:18:48.705 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/nvme1n1 2024-01-13T08:18:48.705 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-13T08:18:48.706 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 193465 Links: 1 Device type: 103,1 2024-01-13T08:18:48.706 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-13T08:18:48.706 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:nvme_device_t:s0 2024-01-13T08:18:48.706 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-01-13 08:17:28.824682748 +0000 2024-01-13T08:18:48.706 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-01-13 08:15:40.237192217 +0000 2024-01-13T08:18:48.706 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-01-13 08:15:40.237192217 +0000 2024-01-13T08:18:48.706 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2024-01-13T08:18:48.706 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-01-13T08:18:48.771 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-01-13T08:18:48.772 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-01-13T08:18:48.772 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000280019 s, 1.8 MB/s 2024-01-13T08:18:48.773 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-01-13T08:18:48.834 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/nvme2n1 2024-01-13T08:18:48.892 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/nvme2n1 2024-01-13T08:18:48.893 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-13T08:18:48.893 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 197669 Links: 1 Device type: 103,2 2024-01-13T08:18:48.893 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-13T08:18:48.893 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:nvme_device_t:s0 2024-01-13T08:18:48.893 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-01-13 08:17:28.835682494 +0000 2024-01-13T08:18:48.893 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-01-13 08:15:40.386188774 +0000 2024-01-13T08:18:48.893 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-01-13 08:15:40.386188774 +0000 2024-01-13T08:18:48.893 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2024-01-13T08:18:48.893 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-01-13T08:18:48.959 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-01-13T08:18:48.959 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-01-13T08:18:48.959 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000238829 s, 2.1 MB/s 2024-01-13T08:18:48.960 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-01-13T08:18:49.019 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/nvme3n1 2024-01-13T08:18:49.077 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/nvme3n1 2024-01-13T08:18:49.077 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-13T08:18:49.077 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 197761 Links: 1 Device type: 103,3 2024-01-13T08:18:49.077 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-13T08:18:49.077 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:nvme_device_t:s0 2024-01-13T08:18:49.077 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-01-13 08:17:28.845682263 +0000 2024-01-13T08:18:49.077 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-01-13 08:15:40.535185330 +0000 2024-01-13T08:18:49.077 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-01-13 08:15:40.535185330 +0000 2024-01-13T08:18:49.077 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2024-01-13T08:18:49.078 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-01-13T08:18:49.143 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-01-13T08:18:49.144 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-01-13T08:18:49.144 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000185975 s, 2.8 MB/s 2024-01-13T08:18:49.145 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-01-13T08:18:49.203 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/nvme4n1 2024-01-13T08:18:49.262 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/nvme4n1 2024-01-13T08:18:49.262 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-13T08:18:49.262 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 197873 Links: 1 Device type: 103,4 2024-01-13T08:18:49.262 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-13T08:18:49.262 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:nvme_device_t:s0 2024-01-13T08:18:49.262 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-01-13 08:17:28.855682031 +0000 2024-01-13T08:18:49.262 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-01-13 08:15:40.685181864 +0000 2024-01-13T08:18:49.262 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-01-13 08:15:40.685181864 +0000 2024-01-13T08:18:49.262 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2024-01-13T08:18:49.263 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-01-13T08:18:49.328 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-01-13T08:18:49.329 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-01-13T08:18:49.329 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000257421 s, 2.0 MB/s 2024-01-13T08:18:49.330 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-01-13T08:18:49.388 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-01-13T08:18:49.388 DEBUG:teuthology.orchestra.run.smithi131:> dd if=/scratch_devs of=/dev/stdout 2024-01-13T08:18:49.410 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-01-13T08:18:49.410 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/nvme1n1 2024-01-13T08:18:49.473 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/nvme1n1 2024-01-13T08:18:49.473 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-13T08:18:49.473 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 6h/6d Inode: 197763 Links: 1 Device type: 103,1 2024-01-13T08:18:49.473 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-13T08:18:49.473 INFO:teuthology.orchestra.run.smithi131.stdout:Context: system_u:object_r:nvme_device_t:s0 2024-01-13T08:18:49.473 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2024-01-13 08:18:26.516052892 +0000 2024-01-13T08:18:49.473 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2024-01-13 08:15:41.899519332 +0000 2024-01-13T08:18:49.473 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2024-01-13 08:15:41.899519332 +0000 2024-01-13T08:18:49.473 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: - 2024-01-13T08:18:49.474 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-01-13T08:18:49.544 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2024-01-13T08:18:49.544 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2024-01-13T08:18:49.544 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000288058 s, 1.8 MB/s 2024-01-13T08:18:49.545 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-01-13T08:18:49.606 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/nvme2n1 2024-01-13T08:18:49.666 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/nvme2n1 2024-01-13T08:18:49.666 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-13T08:18:49.666 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 6h/6d Inode: 194122 Links: 1 Device type: 103,2 2024-01-13T08:18:49.666 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-13T08:18:49.666 INFO:teuthology.orchestra.run.smithi131.stdout:Context: system_u:object_r:nvme_device_t:s0 2024-01-13T08:18:49.666 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2024-01-13 08:18:26.528052640 +0000 2024-01-13T08:18:49.667 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2024-01-13 08:15:42.049516175 +0000 2024-01-13T08:18:49.667 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2024-01-13 08:15:42.049516175 +0000 2024-01-13T08:18:49.667 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: - 2024-01-13T08:18:49.667 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-01-13T08:18:49.735 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2024-01-13T08:18:49.735 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2024-01-13T08:18:49.736 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000233043 s, 2.2 MB/s 2024-01-13T08:18:49.736 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-01-13T08:18:49.796 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/nvme3n1 2024-01-13T08:18:49.855 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/nvme3n1 2024-01-13T08:18:49.855 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-13T08:18:49.855 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 6h/6d Inode: 197996 Links: 1 Device type: 103,3 2024-01-13T08:18:49.855 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-13T08:18:49.855 INFO:teuthology.orchestra.run.smithi131.stdout:Context: system_u:object_r:nvme_device_t:s0 2024-01-13T08:18:49.855 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2024-01-13 08:18:26.540052387 +0000 2024-01-13T08:18:49.856 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2024-01-13 08:15:42.199513017 +0000 2024-01-13T08:18:49.856 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2024-01-13 08:15:42.199513017 +0000 2024-01-13T08:18:49.856 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: - 2024-01-13T08:18:49.856 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-01-13T08:18:49.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:49 smithi131 conmon[38024]: audit 2024-01-13T08:18:48.580367+0000 mon.smithi099 (mon.0) 202 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:49.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:49 smithi131 conmon[38024]: audit 2024-01-13T08:18:48.580981+0000 mon.smithi099 (mon.0) 203 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:18:49.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:49 smithi131 conmon[38024]: audit 2024-01-13T08:18:48.581541+0000 mon.smithi099 (mon.0) 204 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T08:18:49.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:49 smithi131 conmon[38024]: cephadm 2024-01-13T08:18:48.582276+0000 mgr.smithi099.oodult (mgr.14180) 26 : cephadm [INF] Updating smithi099:/etc/ceph/ceph.conf 2024-01-13T08:18:49.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:49 smithi131 conmon[38024]: cephadm 2024-01-13T08:18:48.582510+0000 mgr.smithi099.oodult (mgr.14180) 27 : cephadm [INF] Updating smithi131:/etc/ceph/ceph.conf 2024-01-13T08:18:49.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:49 smithi131 conmon[38024]: audit 2024-01-13T08:18:48.603751+0000 mon.smithi099 (mon.0) 205 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:49.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:49 smithi131 conmon[38024]: audit 2024-01-13T08:18:48.606985+0000 mon.smithi099 (mon.0) 206 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:49.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:49 smithi131 conmon[38024]: audit 2024-01-13T08:18:48.610006+0000 mon.smithi099 (mon.0) 207 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:49.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:49 smithi131 conmon[38024]: cephadm 2024-01-13T08:18:48.612474+0000 mgr.smithi099.oodult (mgr.14180) 28 2024-01-13T08:18:49.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:49 smithi131 conmon[38024]: : cephadm [INF] Reconfiguring mon.smithi099 (unknown last config time)... 2024-01-13T08:18:49.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:49 smithi131 conmon[38024]: audit 2024-01-13T08:18:48.612696+0000 mon.smithi099 (mon.0) 208 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-13T08:18:49.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:49 smithi131 conmon[38024]: audit 2024-01-13T08:18:48.613291+0000 mon.smithi099 (mon.0) 209 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-13T08:18:49.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:49 smithi131 conmon[38024]: audit 2024-01-13T08:18:48.613803+0000 mon.smithi099 (mon.0) 210 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:18:49.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:49 smithi131 conmon[38024]: cephadm 2024-01-13T08:18:48.614277+0000 mgr.smithi099.oodult (mgr.14180) 29 : cephadm [INF] Reconfiguring daemon mon.smithi099 on smithi099 2024-01-13T08:18:49.923 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2024-01-13T08:18:49.923 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2024-01-13T08:18:49.923 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000245585 s, 2.1 MB/s 2024-01-13T08:18:49.925 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-01-13T08:18:49.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:49 smithi099 conmon[33313]: audit 2024-01-13T08:18:48.580367+0000 mon.smithi099 (mon.0) 202 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:49.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:49 smithi099 conmon[33313]: audit 2024-01-13T08:18:48.580981+0000 mon.smithi099 (mon.0) 203 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:18:49.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:49 smithi099 conmon[33313]: audit 2024-01-13T08:18:48.581541+0000 mon.smithi099 (mon.0) 204 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T08:18:49.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:49 smithi099 conmon[33313]: cephadm 2024-01-13T08:18:48.582276+0000 mgr.smithi099.oodult (mgr.14180) 26 : cephadm [INF] Updating smithi099:/etc/ceph/ceph.conf 2024-01-13T08:18:49.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:49 smithi099 conmon[33313]: cephadm 2024-01-13T08:18:48.582510+0000 mgr.smithi099.oodult (mgr.14180) 27 : cephadm [INF] Updating smithi131:/etc/ceph/ceph.conf 2024-01-13T08:18:49.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:49 smithi099 conmon[33313]: audit 2024-01-13T08:18:48.603751+0000 mon.smithi099 (mon.0) 205 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:49.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:49 smithi099 conmon[33313]: audit 2024-01-13T08:18:48.606985+0000 mon.smithi099 (mon 2024-01-13T08:18:49.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:49 smithi099 conmon[33313]: .0) 206 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:49.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:49 smithi099 conmon[33313]: audit 2024-01-13T08:18:48.610006+0000 mon.smithi099 (mon.0) 207 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:49.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:49 smithi099 conmon[33313]: cephadm 2024-01-13T08:18:48.612474+0000 mgr.smithi099.oodult (mgr.14180) 28 : cephadm [INF] Reconfiguring mon.smithi099 (unknown last config time)... 2024-01-13T08:18:49.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:49 smithi099 conmon[33313]: audit 2024-01-13T08:18:48.612696+0000 mon.smithi099 (mon.0) 208 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-13T08:18:49.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:49 smithi099 conmon[33313]: audit 2024-01-13T08:18:48.613291+0000 mon.smithi099 (mon.0) 209 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-13T08:18:49.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:49 smithi099 conmon[33313]: audit 2024-01-13T08:18:48.613803+0000 mon.smithi099 (mon.0) 210 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:18:49.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:49 smithi099 conmon[33313]: cephadm 2024-01-13T08:18:48.614277+0000 mgr.smithi099.oodult (mgr.14180) 29 : cephadm [INF] Reconfiguring daemon mon.smithi099 on smithi099 2024-01-13T08:18:49.983 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/nvme4n1 2024-01-13T08:18:50.042 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/nvme4n1 2024-01-13T08:18:50.042 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-01-13T08:18:50.042 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 6h/6d Inode: 194149 Links: 1 Device type: 103,4 2024-01-13T08:18:50.042 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-01-13T08:18:50.043 INFO:teuthology.orchestra.run.smithi131.stdout:Context: system_u:object_r:nvme_device_t:s0 2024-01-13T08:18:50.043 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2024-01-13 08:18:26.552052135 +0000 2024-01-13T08:18:50.043 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2024-01-13 08:15:42.348509881 +0000 2024-01-13T08:18:50.043 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2024-01-13 08:15:42.348509881 +0000 2024-01-13T08:18:50.043 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: - 2024-01-13T08:18:50.043 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-01-13T08:18:50.108 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2024-01-13T08:18:50.108 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2024-01-13T08:18:50.109 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000233574 s, 2.2 MB/s 2024-01-13T08:18:50.109 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-01-13T08:18:50.168 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph orch apply osd --all-available-devices 2024-01-13T08:18:50.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:50 smithi131 conmon[38024]: cluster 2024-01-13T08:18:49.104798+0000 mgr.smithi099.oodult (mgr.14180 2024-01-13T08:18:50.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:50 smithi131 conmon[38024]: ) 30 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:18:50.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:50 smithi131 conmon[38024]: audit 2024-01-13T08:18:49.798511+0000 mon.smithi099 (mon.0) 211 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:50.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:50 smithi131 conmon[38024]: audit 2024-01-13T08:18:49.799309+0000 mon.smithi099 (mon.0) 212 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi099.oodult", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-13T08:18:50.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:50 smithi131 conmon[38024]: audit 2024-01-13T08:18:49.800004+0000 mon.smithi099 (mon.0) 213 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-13T08:18:50.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:50 smithi131 conmon[38024]: audit 2024-01-13T08:18:49.800506+0000 mon.smithi099 (mon.0) 214 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:18:50.950 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:50 smithi099 conmon[33313]: cluster 2024-01-13T08:18:49.104798+0000 mgr.smithi099.oodult (mgr. 2024-01-13T08:18:50.950 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:50 smithi099 conmon[33313]: 14180) 30 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:18:50.951 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:50 smithi099 conmon[33313]: audit 2024-01-13T08:18:49.798511+0000 mon.smithi099 (mon.0) 211 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:50.951 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:50 smithi099 conmon[33313]: audit 2024-01-13T08:18:49.799309+0000 mon.smithi099 (mon.0) 212 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi099.oodult", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-13T08:18:50.951 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:50 smithi099 conmon[33313]: audit 2024-01-13T08:18:49.800004+0000 mon.smithi099 (mon.0) 213 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-13T08:18:50.951 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:50 smithi099 conmon[33313]: audit 2024-01-13T08:18:49.800506+0000 mon.smithi099 (mon.0) 214 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:18:51.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:51 smithi099 conmon[33313]: cephadm 2024-01-13T08:18:49.799074+0000 2024-01-13T08:18:51.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:51 smithi099 conmon[33313]: mgr.smithi099.oodult (mgr.14180) 31 : cephadm [INF] Reconfiguring mgr.smithi099.oodult (unknown last config time)... 2024-01-13T08:18:51.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:51 smithi099 conmon[33313]: cephadm 2024-01-13T08:18:49.800987+0000 mgr.smithi099.oodult (mgr.14180) 32 : cephadm [INF] Reconfiguring daemon mgr.smithi099.oodult on smithi099 2024-01-13T08:18:51.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:51 smithi099 conmon[33313]: audit 2024-01-13T08:18:51.002859+0000 mon.smithi099 (mon.0) 215 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:51.812 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:51 smithi131 conmon[38024]: cephadm 2024-01-13T08:18:49.799074+0000 mgr.smithi099.oodult (mgr.14180) 31 : cephadm 2024-01-13T08:18:51.812 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:51 smithi131 conmon[38024]: [INF] Reconfiguring mgr.smithi099.oodult (unknown last config time)... 2024-01-13T08:18:51.812 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:51 smithi131 conmon[38024]: cephadm 2024-01-13T08:18:49.800987+0000 mgr.smithi099.oodult (mgr.14180) 32 : cephadm [INF] Reconfiguring daemon mgr.smithi099.oodult on smithi099 2024-01-13T08:18:51.813 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:51 smithi131 conmon[38024]: audit 2024-01-13T08:18:51.002859+0000 mon.smithi099 (mon.0) 215 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:52.135 INFO:teuthology.orchestra.run.smithi131.stdout:Scheduled osd.all-available-devices update... 2024-01-13T08:18:52.716 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-01-13T08:18:52.716 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:18:52.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:52 smithi099 conmon[33313]: cephadm 2024-01-13T08:18:51.003504+0000 mgr.smithi099.oodult (mgr.14180) 33 : cephadm [INF] Reconfiguring alertmanager.smithi099 (dependencies changed)... 2024-01-13T08:18:52.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:52 smithi099 conmon[33313]: cephadm 2024-01-13T08:18:51.009694+0000 mgr.smithi099.oodult (mgr.14180) 34 : cephadm [INF] Reconfiguring daemon alertmanager.smithi099 on smithi099 2024-01-13T08:18:52.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:52 smithi099 conmon[33313]: cluster 2024-01-13T08:18:51.105064+0000 mgr.smithi099.oodult (mgr.14180) 35 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:18:52.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:52 smithi099 conmon[33313]: audit 2024-01-13T08:18:52.139116+0000 mon.smithi099 (mon.0) 216 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:52.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:52 smithi099 conmon[33313]: audit 2024-01-13T08:18:52.257701+0000 mon.smithi099 (mon.0) 217 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:52.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:52 smithi131 conmon[38024]: cephadm 2024-01-13T08:18:51.003504+0000 mgr.smithi099.oodult (mgr.14180) 33 : cephadm [INF] Reconfiguring alertmanager.smithi099 (dependencies changed)... 2024-01-13T08:18:52.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:52 smithi131 conmon[38024]: cephadm 2024-01-13T08:18:51.009694+0000 mgr.smithi099.oodult (mgr.14180) 34 : cephadm [INF] Reconfiguring daemon alertmanager.smithi099 on smithi099 2024-01-13T08:18:52.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:52 smithi131 conmon[38024]: cluster 2024-01-13T08:18:51.105064+0000 mgr.smithi099.oodult (mgr.14180) 35 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:18:52.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:52 smithi131 conmon[38024]: audit 2024-01-13T08:18:52.139116+0000 mon.smithi099 (mon.0) 216 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:52.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:52 smithi131 conmon[38024]: audit 2024-01-13T08:18 2024-01-13T08:18:52.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:52 smithi131 conmon[38024]: :52.257701+0000 mon.smithi099 (mon.0) 217 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:53.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:53 smithi131 conmon[38024]: audit 2024-01-13T08:18:52.132484+0000 mgr.smithi099.oodult (mgr.14180) 36 : audit [DBG] from='client.24101 -' entity='client.admin' cmd=[{"prefix": "orch apply osd", "all_available_devices": true, "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T08:18:53.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:53 smithi131 conmon[38024]: cephadm 2024-01-13T08:18:52.133822+0000 mgr.smithi099.oodult (mgr.14180) 37 : cephadm [INF] Marking host: smithi099 for OSDSpec preview refresh. 2024-01-13T08:18:53.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:53 smithi131 conmon[38024]: cephadm 2024-01-13T08:18:52.133880+0000 mgr.smithi099.oodult (mgr.14180) 38 : cephadm [INF] Marking host: smithi131 for OSDSpec preview refresh. 2024-01-13T08:18:53.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:53 smithi131 conmon[38024]: cephadm 2024-01-13T08:18:52.134121+0000 mgr.smithi099.oodult (mgr.14180) 39 : cephadm [INF] Saving service osd.all-available-devices spec with placement * 2024-01-13T08:18:53.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:53 smithi099 conmon[33313]: audit 2024-01-13T08:18:52.132484+0000 mgr.smithi099.oodult (mgr.14180 2024-01-13T08:18:53.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:53 smithi099 conmon[33313]: ) 36 : audit [DBG] from='client.24101 -' entity='client.admin' cmd=[{"prefix": "orch apply osd", "all_available_devices": true, "target": ["mon-mgr", ""]}]: dispatch 2024-01-13T08:18:53.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:53 smithi099 conmon[33313]: cephadm 2024-01-13T08:18:52.133822+0000 mgr.smithi099.oodult (mgr.14180) 37 : cephadm [INF] Marking host: smithi099 for OSDSpec preview refresh. 2024-01-13T08:18:53.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:53 smithi099 conmon[33313]: cephadm 2024-01-13T08:18:52.133880+0000 mgr.smithi099.oodult (mgr.14180) 38 : cephadm [INF] Marking host: smithi131 for OSDSpec preview refresh. 2024-01-13T08:18:53.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:53 smithi099 conmon[33313]: cephadm 2024-01-13T08:18:52.134121+0000 mgr.smithi099.oodult (mgr.14180) 39 : cephadm [INF] Saving service osd.all-available-devices spec with placement * 2024-01-13T08:18:54.798 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:54 smithi099 conmon[33313]: cluster 2024-01-13T08:18:53.105358+0000 mgr.smithi099.oodult (mgr.14180) 40 : 2024-01-13T08:18:54.798 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:54 smithi099 conmon[33313]: cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:18:54.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:54 smithi131 conmon[38024]: cluster 2024-01-13T08:18:53.105358+0000 mgr.smithi099.oodult (mgr.14180) 2024-01-13T08:18:54.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:54 smithi131 conmon[38024]: 40 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:18:55.154 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:18:55.885 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:55 smithi099 conmon[33313]: audit 2024-01-13T08:18:55.158172+0000 mon.smithi099 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.99:0/960706055' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:18:55.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:55 smithi131 conmon[38024]: audit 2024-01-13T08:18:55.158172+0000 mon.smithi099 (mon.0 2024-01-13T08:18:55.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:55 smithi131 conmon[38024]: ) 218 : audit [DBG] from='client.? 172.21.15.99:0/960706055' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:18:56.141 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2024-01-13T08:18:56.882 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:56 smithi099 conmon[33313]: cluster 2024-01-13T08:18:55.105563+0000 mgr.smithi099.oodult (mgr.14180 2024-01-13T08:18:56.882 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:56 smithi099 conmon[33313]: ) 41 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:18:56.883 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:56 smithi099 conmon[33313]: audit 2024-01-13T08:18:56.089393+0000 mon.smithi099 (mon.0) 219 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:56.883 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:56 smithi099 conmon[33313]: audit 2024-01-13T08:18:56.090234+0000 mon.smithi099 (mon.0) 220 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi099", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-13T08:18:56.883 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:56 smithi099 conmon[33313]: audit 2024-01-13T08:18:56.090859+0000 mon.smithi099 (mon.0) 221 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:18:56.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:56 smithi131 conmon[38024]: cluster 2024-01-13T08:18:55.105563+0000 mgr.smithi099.oodult (mgr.14180) 41 : cluster [DBG] 2024-01-13T08:18:56.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:56 smithi131 conmon[38024]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:18:56.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:56 smithi131 conmon[38024]: audit 2024-01-13T08:18:56.089393+0000 mon.smithi099 (mon.0) 219 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:56.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:56 smithi131 conmon[38024]: audit 2024-01-13T08:18:56.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:56 smithi131 conmon[38024]: 2024-01-13T08:18:56.090234+0000 mon.smithi099 (mon.0) 220 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi099", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-13T08:18:56.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:56 smithi131 conmon[38024]: audit 2024-01-13T08:18:56.090859+0000 mon.smithi099 (mon 2024-01-13T08:18:56.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:56 smithi131 conmon[38024]: .0) 221 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:18:57.143 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:18:57.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:57 smithi131 conmon[38024]: cephadm 2024-01-13T08:18:56.089970+0000 mgr.smithi099.oodult (mgr.14180) 2024-01-13T08:18:57.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:57 smithi131 conmon[38024]: 42 : cephadm [INF] Reconfiguring crash.smithi099 (monmap changed)... 2024-01-13T08:18:57.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:57 smithi131 conmon[38024]: cephadm 2024-01-13T08:18:56.091321+0000 mgr.smithi099.oodult (mgr.14180) 43 2024-01-13T08:18:57.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:57 smithi131 conmon[38024]: : cephadm [INF] Reconfiguring daemon crash.smithi099 on smithi099 2024-01-13T08:18:57.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:57 smithi131 conmon[38024]: audit 2024-01 2024-01-13T08:18:57.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:57 smithi131 conmon[38024]: -13T08:18:57.420940+0000 mon.smithi099 (mon.0) 222 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:57.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:57 smithi131 conmon[38024]: 2024-01-13T08:18:57.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:57 smithi099 conmon[33313]: cephadm 2024-01-13T08:18:56.089970+0000 mgr.smithi099.oodult (mgr.14180) 42 : cephadm [INF] Reconfiguring crash.smithi099 (monmap changed)... 2024-01-13T08:18:57.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:57 smithi099 conmon[33313]: cephadm 2024-01-13T08:18:56.091321+0000 mgr.smithi099.oodult (mgr.14180) 43 : cephadm [INF] Reconfiguring daemon crash.smithi099 on smithi099 2024-01-13T08:18:57.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:57 smithi099 conmon[33313]: audit 2024-01-13T08:18:57.420940+0000 mon.smithi099 (mon.0) 222 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:18:58.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:58 smithi131 conmon[38024]: cluster 2024-01-13T08:18:57.105767+0000 mgr.smithi099.oodult (mgr.14180) 2024-01-13T08:18:58.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:58 smithi131 conmon[38024]: 44 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:18:58.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:58 smithi131 conmon[38024]: cephadm 2024-01-13T08:18:57.421762+0000 mgr.smithi099.oodult ( 2024-01-13T08:18:58.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:58 smithi131 conmon[38024]: mgr.14180) 45 : cephadm [INF] Reconfiguring grafana.smithi099 (dependencies changed)... 2024-01-13T08:18:58.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:58 smithi131 conmon[38024]: cephadm 2024-01-13T08:18:58.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:58 smithi131 conmon[38024]: 2024-01-13T08:18:57.464018+0000 mgr.smithi099.oodult (mgr.14180) 46 : cephadm [INF] 2024-01-13T08:18:58.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:18:58 smithi131 conmon[38024]: Reconfiguring daemon grafana.smithi099 on smithi099 2024-01-13T08:18:58.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:58 smithi099 conmon[33313]: cluster 2024-01-13T08:18:57.105767+0000 mgr.smithi099.oodult (mgr.14180) 44 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:18:58.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:58 smithi099 conmon[33313]: 2024-01-13T08:18:58.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:58 smithi099 conmon[33313]: cephadm 2024-01-13T08:18:57.421762+0000 mgr.smithi099.oodult (mgr.14180) 45 : cephadm [INF] Reconfiguring grafana.smithi099 (dependencies changed)... 2024-01-13T08:18:58.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:18:58 smithi099 conmon[33313]: cephadm 2024-01-13T08:18:57.464018+0000 mgr.smithi099.oodult (mgr.14180) 46 : cephadm [INF] Reconfiguring daemon grafana.smithi099 on smithi099 2024-01-13T08:18:59.671 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:19:00.597 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2024-01-13T08:19:00.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:00 smithi131 conmon[38024]: cluster 2024-01-13T08:18:59.106096+0000 mgr.smithi099.oodult (mgr.14180) 47 : 2024-01-13T08:19:00.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:00 smithi131 conmon[38024]: cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:00.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:00 smithi131 conmon[38024]: audit 2024-01-13T08:18:59.649607+0000 mon.smithi099 (mon.0) 223 : audit [DBG] from='client.? 172.21.15.99:0/2830614940' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:19:00.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:00 smithi131 conmon[38024]: 2024-01-13T08:19:00.912 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:00 smithi099 conmon[33313]: cluster 2024-01-13T08:18:59.106096+0000 mgr.smithi099.oodult (mgr.14180) 47 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:00.912 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:00 smithi099 conmon[33313]: audit 2024-01-13T08:18:59.649607+0000 mon.smithi099 (mon.0) 223 : audit [DBG] from='client.? 172.21.15.99:0/2830614940' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:19:01.598 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:19:02.211 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:01 smithi099 conmon[33313]: cluster 2024-01-13T08:19:01.106314+0000 mgr.smithi099.oodult (mgr 2024-01-13T08:19:02.211 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:01 smithi099 conmon[33313]: .14180) 48 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:02.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:01 smithi131 conmon[38024]: cluster 2024-01-13T08:19:01.106314+0000 mgr.smithi099.oodult (mgr. 2024-01-13T08:19:02.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:01 smithi131 conmon[38024]: 14180) 48 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:03.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:03 smithi099 conmon[33313]: audit 2024-01-13T08:19:02.503705+0000 mon.smithi099 (mon.0) 224 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:19:03.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:03 smithi099 conmon[33313]: cephadm 2024-01-13T08:19:02.504472+0000 mgr.smithi099.oodult (mgr.14180) 49 : cephadm [INF] Reconfiguring prometheus.smithi099 (dependencies changed)... 2024-01-13T08:19:03.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:03 smithi099 conmon[33313]: cephadm 2024-01-13T08:19:02.507588+0000 mgr.smithi099.oodult (mgr.14180) 50 : cephadm [INF] Reconfiguring daemon prometheus.smithi099 on smithi099 2024-01-13T08:19:03.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:03 smithi131 conmon[38024]: audit 2024-01-13T08:19:02.503705+0000 mon.smithi099 (mon.0) 2024-01-13T08:19:03.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:03 smithi131 conmon[38024]: 224 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:19:03.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:03 smithi131 conmon[38024]: cephadm 2024-01-13T08:19:02.504472+0000 mgr.smithi099.oodult (mgr.14180) 49 : cephadm 2024-01-13T08:19:03.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:03 smithi131 conmon[38024]: [INF] Reconfiguring prometheus.smithi099 (dependencies changed)... 2024-01-13T08:19:03.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:03 smithi131 conmon[38024]: cephadm 2024-01-13T08:19:02.507588+0000 mgr.smithi099.oodult (mgr. 2024-01-13T08:19:03.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:03 smithi131 conmon[38024]: 14180) 50 : cephadm [INF] Reconfiguring daemon prometheus.smithi099 on smithi099 2024-01-13T08:19:04.058 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:19:04.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:04 smithi131 conmon[38024]: cluster 2024-01-13T08:19:03.106547+0000 mgr.smithi099.oodult ( 2024-01-13T08:19:04.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:04 smithi131 conmon[38024]: mgr.14180) 51 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:04.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:04 smithi131 conmon[38024]: audit 2024-01-13T08:19:04.061878+0000 2024-01-13T08:19:04.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:04 smithi131 conmon[38024]: mon.smithi099 (mon.0) 225 : audit [DBG] from='client.? 172.21.15.99:0/4234501095' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:19:04.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:04 smithi099 conmon[33313]: cluster 2024-01-13T08:19:03.106547+0000 mgr.smithi099.oodult (mgr.14180) 2024-01-13T08:19:04.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:04 smithi099 conmon[33313]: 51 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:04.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:04 smithi099 conmon[33313]: audit 2024-01-13T08:19:04.061878+0000 mon.smithi099 (mon.0) 225 : audit [DBG] from='client.? 172.21.15.99:0/4234501095' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:19:05.277 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2024-01-13T08:19:06.278 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:19:06.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:06 smithi099 conmon[33313]: cluster 2024-01-13T08:19:05.106800+0000 mgr.smithi099.oodult (mgr.14180) 52 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:06.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:06 smithi131 conmon[38024]: cluster 2024-01-13T08:19:05.106800+0000 mgr.smithi099.oodult (mgr.14180) 52 : 2024-01-13T08:19:06.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:06 smithi131 conmon[38024]: cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:08.436 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:19:08.714 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:08 smithi099 conmon[33313]: cluster 2024-01-13T08:19:07.107022+0000 2024-01-13T08:19:08.714 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:08 smithi099 conmon[33313]: mgr.smithi099.oodult (mgr.14180) 53 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:08.715 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:08 smithi099 conmon[33313]: audit 2024-01-13T08:19:08.439285+0000 mon.smithi099 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.99:0/1696956956' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:19:08.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:08 smithi131 conmon[38024]: cluster 2024-01-13T08:19:07.107022+0000 mgr.smithi099.oodult (mgr.14180) 53 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:08.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:08 smithi131 conmon[38024]: audit 2024-01-13T08:19:08.439285+0000 mon.smithi099 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.99:0/1696956956' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:19:09.263 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2024-01-13T08:19:09.796 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:09 smithi131 conmon[38024]: audit 2024-01-13T08:19:08.616662+0000 mon.smithi099 (mon.0) 227 : 2024-01-13T08:19:09.796 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:09 smithi131 conmon[38024]: audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:19:09.797 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:09 smithi131 conmon[38024]: cephadm 2024-01-13T08:19:08.617194+0000 mgr.smithi099.oodult (mgr.14180) 54 : cephadm [INF] Reconfiguring crash.smithi131 (monmap changed)... 2024-01-13T08:19:09.797 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:09 smithi131 conmon[38024]: audit 2024-01-13T08:19:08.617407+0000 mon.smithi099 (mon.0) 228 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi131", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-13T08:19:09.797 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:09 smithi131 conmon[38024]: audit 2024-01-13T08:19:08.618005+0000 mon.smithi099 (mon.0) 229 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:19:09.797 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:09 smithi131 conmon[38024]: cephadm 2024-01-13T08:19:08.618474+0000 mgr.smithi099.oodult (mgr.14180) 55 : cephadm [INF] Reconfiguring daemon crash.smithi131 on smithi131 2024-01-13T08:19:09.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:09 smithi099 conmon[33313]: audit 2024-01-13T08:19:08.616662+0000 mon.smithi099 (mon.0) 227 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:19:09.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:09 smithi099 conmon[33313]: cephadm 2024-01-13T08:19:08.617194+0000 mgr.smithi099.oodult (mgr.14180) 54 : cephadm [INF] Reconfiguring crash.smithi131 (monmap changed)... 2024-01-13T08:19:09.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:09 smithi099 conmon[33313]: audit 2024-01-13T 2024-01-13T08:19:09.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:09 smithi099 conmon[33313]: 08:19:08.617407+0000 2024-01-13T08:19:09.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:09 smithi099 conmon[33313]: mon.smithi099 (mon.0) 228 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi131", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-13T08:19:09.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:09 smithi099 conmon[33313]: audit 2024-01-13T 2024-01-13T08:19:09.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:09 smithi099 conmon[33313]: 08:19:08.618005+0000 mon.smithi099 (mon. 2024-01-13T08:19:09.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:09 smithi099 conmon[33313]: 0) 229 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:19:09.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:09 smithi099 conmon[33313]: cephadm 2024-01- 2024-01-13T08:19:09.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:09 smithi099 conmon[33313]: 13T08:19:08.618474+0000 mgr.smithi099.oodult (mgr.14180) 2024-01-13T08:19:09.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:09 smithi099 conmon[33313]: 55 : cephadm [INF] Reconfiguring daemon crash.smithi131 on smithi131 2024-01-13T08:19:10.264 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:19:10.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:10 smithi131 conmon[38024]: cluster 2024-01-13T08:19:09.107377+0000 mgr.smithi099.oodult (mgr.14180) 56 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:10.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:10 smithi131 conmon[38024]: 2024-01-13T08:19:10.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:10 smithi131 conmon[38024]: audit 2024-01-13T08:19:09.840017+0000 mon.smithi099 (mon.0) 230 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:19:10.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:10 smithi131 conmon[38024]: audit 2024-01-13T08:19:09.841042+0000 mon.smithi099 (mon.0) 231 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi131.qnazik", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-13T08:19:10.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:10 smithi131 conmon[38024]: audit 2024-01-13T08:19:09.841868+0000 mon.smithi099 (mon.0) 232 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-13T08:19:10.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:10 smithi131 conmon[38024]: audit 2024-01-13T08:19:09.842558+0000 mon.smithi099 (mon.0) 233 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:19:10.916 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:10 smithi099 conmon[33313]: cluster 2024-01-13T08:19:09.107377+0000 mgr.smithi099.oodult (mgr.14180) 56 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:10.916 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:10 smithi099 conmon[33313]: audit 2024-01-13T08:19:09.840017+0000 mon.smithi099 (mon.0) 230 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:19:10.916 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:10 smithi099 conmon[33313]: audit 2024-01-13T08:19:09.841042+0000 mon.smithi099 (mon.0) 231 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi131.qnazik", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-13T08:19:10.916 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:10 smithi099 conmon[33313]: audit 2024-01-13T08:19:09.841868+0000 mon.smithi099 (mon.0) 232 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-13T08:19:10.916 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:10 smithi099 conmon[33313]: audit 2024-01-13T08:19:09.842558+0000 mon.smithi099 (mon.0) 233 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:19:11.830 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:11 smithi099 conmon[33313]: cephadm 2024-01-13T08:19:09.840748+0000 mgr.smithi099.oodult (mgr. 2024-01-13T08:19:11.830 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:11 smithi099 conmon[33313]: 14180) 57 : cephadm [INF] Reconfiguring mgr.smithi131.qnazik (monmap changed)... 2024-01-13T08:19:11.831 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:11 smithi099 conmon[33313]: cephadm 2024-01-13T08:19:09.843147+0000 mgr.smithi099.oodult (mgr.14180) 58 : cephadm [INF] Reconfiguring daemon mgr.smithi131.qnazik on smithi131 2024-01-13T08:19:11.831 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:11 smithi099 conmon[33313]: audit 2024-01-13T08:19:11.061113+0000 mon.smithi099 (mon.0) 234 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:19:11.831 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:11 smithi099 conmon[33313]: audit 2024-01-13T08:19:11.061851+0000 mon.smithi099 (mon.0) 235 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-13T08:19:11.831 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:11 smithi099 conmon[33313]: audit 2024-01-13T08:19:11.062380+0000 mon.smithi099 (mon.0) 236 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-13T08:19:11.831 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:11 smithi099 conmon[33313]: audit 2024-01-13T08:19:11.062846+0000 mon.smithi099 (mon.0) 237 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:19:11.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:11 smithi131 conmon[38024]: cephadm 2024-01-13T08:19:09.840748+0000 mgr.smithi099.oodult (mgr.14180) 57 : cephadm [INF] Reconfiguring mgr.smithi131.qnazik (monmap changed)... 2024-01-13T08:19:11.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:11 smithi131 conmon[38024]: cephadm 2024-01-13T08:19 2024-01-13T08:19:11.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:11 smithi131 conmon[38024]: :09.843147+0000 mgr.smithi099.oodult (mgr.14180) 58 : cephadm [INF] Reconfiguring daemon mgr.smithi131.qnazik on smithi131 2024-01-13T08:19:11.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:11 smithi131 conmon[38024]: audit 2024-01-13T08:19:11.061113+0000 mon.smithi099 (mon.0) 234 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:19:11.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:11 smithi131 conmon[38024]: audit 2024-01-13T08 2024-01-13T08:19:11.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:11 smithi131 conmon[38024]: :19:11.061851+0000 mon.smithi099 (mon.0) 235 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-13T08:19:11.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:11 smithi131 conmon[38024]: audit 2024-01-13T08:19:11. 2024-01-13T08:19:11.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:11 smithi131 conmon[38024]: 062380+0000 mon.smithi099 (mon.0) 236 : audit 2024-01-13T08:19:11.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:11 smithi131 conmon[38024]: [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-13T08:19:11.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:11 smithi131 conmon[38024]: audit 2024-01-13T08:19:11.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:11 smithi131 conmon[38024]: 2024-01-13T08:19:11.062846+0000 mon.smithi099 (mon 2024-01-13T08:19:11.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:11 smithi131 conmon[38024]: .0) 237 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:19:12.142 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:19:12.766 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: cephadm 2024-01-13T08:19:11 2024-01-13T08:19:12.767 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: .061624+0000 mgr.smithi099.oodult (mgr.14180) 59 : cephadm [INF] Reconfiguring mon.smithi131 (monmap changed)... 2024-01-13T08:19:12.767 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: cephadm 2024- 2024-01-13T08:19:12.767 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: 01-13T08:19:11.063284 2024-01-13T08:19:12.767 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: +0000 mgr.smithi099.oodult (mgr.14180) 60 : 2024-01-13T08:19:12.767 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: cephadm [INF] Reconfiguring daemon mon.smithi131 on smithi131 2024-01-13T08:19:12.767 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: cluster 2024-01-13T08:19:12.767 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: 2024-01-13T08: 2024-01-13T08:19:12.767 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: 19:11.107754+0000 mgr.smithi099.oodult (mgr 2024-01-13T08:19:12.767 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: .14180) 61 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:12.767 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: 2024-01-13T08:19:12.767 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: audit 2024-01-13T08:19:12.768 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: 2024-01-13T08:19 2024-01-13T08:19:12.768 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: :12.146314+0000 mon.smithi099 (mon 2024-01-13T08:19:12.768 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: .0) 238 : audit [DBG] from='client.? 172.21.15.99:0/3977312305' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:19:12.768 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: 2024-01-13T08:19:12.768 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: audit 2024 2024-01-13T08:19:12.768 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: -01-13T08:19: 2024-01-13T08:19:12.768 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: 12.363843+0000 mon.smithi099 (mon.0 2024-01-13T08:19:12.768 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: ) 239 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:19:12.768 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: audit 2024-01-13T08:19:12.768 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: 2024-01- 2024-01-13T08:19:12.768 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: 13T08:19:12.366292 2024-01-13T08:19:12.768 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: +0000 mon.smithi099 (mon.0) 240 : 2024-01-13T08:19:12.768 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-13T08:19:12.768 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: audit 2024-01-13T08:19:12.769 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: 2024-01-13T08:19 2024-01-13T08:19:12.769 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: :12.367402+0000 mon.smithi099 (mon. 2024-01-13T08:19:12.769 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: 0) 241 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi099:9093"}]: dispatch 2024-01-13T08:19:12.769 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: 2024-01-13T08:19:12.769 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: audit 2024-01-13T08:19:12.769 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: 2024-01-13T08:19 2024-01-13T08:19:12.769 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: :12.371079+0000 mon.smithi099 (mon. 2024-01-13T08:19:12.769 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: 0) 242 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:19:12.769 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: 2024-01-13T08:19:12.769 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: audit 2024-01 2024-01-13T08:19:12.770 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: -13T08:19:12. 2024-01-13T08:19:12.770 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: 379948+0000 mon.smithi099 (mon.0) 2024-01-13T08:19:12.770 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: 243 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-13T08:19:12.770 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: audit 2024-01-13T08:19:12.770 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: 2024-01-13T08:19 2024-01-13T08:19:12.770 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: :12.382604+0000 mon.smithi099 (mon. 2024-01-13T08:19:12.770 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: 0) 244 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi099:3000"}]: dispatch 2024-01-13T08:19:12.770 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: audit 2024-01-13T08:19:12.770 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: 2024-01-13T08:19 2024-01-13T08:19:12.770 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: :12.386946+0000 mon.smithi099 (mon. 2024-01-13T08:19:12.770 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: 0) 245 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:19:12.770 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: 2024-01-13T08:19:12.771 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: audit 2024-01-13T08 2024-01-13T08:19:12.771 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: :19:12.390921+0000 mon.smithi099 ( 2024-01-13T08:19:12.771 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: mon.0) 246 : audit [DBG] 2024-01-13T08:19:12.771 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-01-13T08:19:12.771 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: audit 2024-01-13T 2024-01-13T08:19:12.771 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: 08:19:12.395548+0000 mon.smithi099 2024-01-13T08:19:12.771 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: (mon.0) 247 : audit [INF] 2024-01-13T08:19:12.771 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi099:9095"}]: dispatch 2024-01-13T08:19:12.771 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: audit 2024-01-13 2024-01-13T08:19:12.771 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: T08:19:12.399826+0000 2024-01-13T08:19:12.771 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: mon.smithi099 (mon.0) 248 : audit 2024-01-13T08:19:12.771 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:19:12.771 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: audit 2024-01-13T08 2024-01-13T08:19:12.771 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: :19:12.401755+0000 mon.smithi099 (mon.0 2024-01-13T08:19:12.772 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:12 smithi099 conmon[33313]: ) 249 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:19:12.772 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2024-01-13T08:19:12.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:12 smithi131 conmon[38024]: cephadm 2024-01-13T08:19:11.061624+0000 mgr.smithi099.oodult (mgr.14180) 59 : cephadm [INF] Reconfiguring mon.smithi131 (monmap changed)... 2024-01-13T08:19:12.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:12 smithi131 conmon[38024]: cephadm 2024-01-13T08:19:11.063284+0000 mgr.smithi099.oodult (mgr.14180) 60 : cephadm [INF] Reconfiguring daemon mon.smithi131 on smithi131 2024-01-13T08:19:12.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:12 smithi131 conmon[38024]: cluster 2024-01-13T08:19:11.107754+0000 mgr.smithi099.oodult (mgr.14180) 61 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:12.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:12 smithi131 conmon[38024]: audit 2024-01-13T08:19:12.146314+0000 mon.smithi099 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.99:0/3977312305' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:19:12.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:12 smithi131 conmon[38024]: audit 2024-01-13T08:19:12.363843+0000 mon.smithi099 (mon.0) 239 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:19:12.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:12 smithi131 conmon[38024]: audit 2024-01-13T08:19:12.366292+0000 mon.smithi099 (mon.0) 240 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-13T08:19:12.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:12 smithi131 conmon[38024]: audit 2024-01-13T08:19:12.367402+0000 mon.smithi099 (mon.0) 241 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi099:9093"}]: dispatch 2024-01-13T08:19:12.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:12 smithi131 conmon[38024]: audit 2024-01-13T08:19:12.371079+0000 mon.smithi099 (mon.0) 242 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:19:12.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:12 smithi131 conmon[38024]: audit 2024-01-13T08:19:12.379948+0000 mon.smithi099 (mon.0) 243 : audit 2024-01-13T08:19:12.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:12 smithi131 conmon[38024]: [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-13T08:19:12.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:12 smithi131 conmon[38024]: audit 2024-01-13T08:19:12.382604+0000 mon.smithi099 (mon.0) 244 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi099:3000"}]: dispatch 2024-01-13T08:19:12.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:12 smithi131 conmon[38024]: audit 2024-01-13T08:19:12.386946+0000 mon.smithi099 (mon.0) 245 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:19:12.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:12 smithi131 conmon[38024]: audit 2024-01-13T08:19:12.390921+0000 mon.smithi099 (mon.0) 246 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-01-13T08:19:12.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:12 smithi131 conmon[38024]: audit 2024-01-13T08:19:12.395548+0000 mon.smithi099 (mon.0) 2024-01-13T08:19:12.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:12 smithi131 conmon[38024]: 247 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi099:9095"}]: dispatch 2024-01-13T08:19:12.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:12 smithi131 conmon[38024]: audit 2024-01-13T08:19:12.399826+0000 2024-01-13T08:19:12.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:12 smithi131 conmon[38024]: mon.smithi099 (mon.0) 248 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:19:12.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:12 smithi131 conmon[38024]: audit 2024- 2024-01-13T08:19:12.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:12 smithi131 conmon[38024]: 01-13T08:19:12.401755+0000 mon.smithi099 2024-01-13T08:19:12.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:12 smithi131 conmon[38024]: (mon.0) 249 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:19:13.774 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:19:13.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:13 smithi131 conmon[38024]: audit 2024-01-13T08:19:12.366632+0000 mgr.smithi099.oodult (mgr.14180) 62 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-13T08:19:13.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:13 smithi131 conmon[38024]: audit 2024-01-13T08:19:12.367629+0000 mgr.smithi099.oodult (mgr.14180) 63 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi099:9093"}]: dispatch 2024-01-13T08:19:13.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:13 smithi131 conmon[38024]: audit 2024-01-13T08:19:12.380478+0000 mgr.smithi099.oodult (mgr.14180) 64 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-13T08:19:13.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:13 smithi131 conmon[38024]: audit 2024-01-13T08:19:12.382938+0000 mgr.smithi099.oodult (mgr.14180) 65 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi099:3000"}]: dispatch 2024-01-13T08:19:13.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:13 smithi131 conmon[38024]: audit 2024-01-13T08:19:12.391335+0000 mgr.smithi099.oodult (mgr.14180) 66 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-01-13T08:19:13.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:13 smithi131 conmon[38024]: audit 2024-01-13T08:19:12.395852+0000 mgr.smithi099.oodult (mgr.14180) 67 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi099:9095"}]: dispatch 2024-01-13T08:19:13.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:13 smithi099 conmon[33313]: audit 2024-01-13T08:19:12.366632+0000 mgr.smithi099.oodult (mgr.14180) 62 : audit [DBG] 2024-01-13T08:19:13.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:13 smithi099 conmon[33313]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-13T08:19:13.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:13 smithi099 conmon[33313]: audit 2024-01-13T08:19:12.367629+0000 mgr.smithi099.oodult (mgr.14180) 63 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi099:9093"}]: dispatch 2024-01-13T08:19:13.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:13 smithi099 conmon[33313]: audit 2024-01-13T08:19:12.380478+0000 mgr.smithi099.oodult (mgr.14180) 64 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-13T08:19:13.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:13 smithi099 conmon[33313]: audit 2024-01-13T08:19:12.382938+0000 mgr.smithi099.oodult (mgr.14180) 65 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi099:3000"}]: dispatch 2024-01-13T08:19:13.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:13 smithi099 conmon[33313]: audit 2024-01-13T08:19:12.391335+0000 mgr.smithi099.oodult (mgr.14180) 66 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-01-13T08:19:13.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:13 smithi099 conmon[33313]: audit 2024-01-13T08:19:12.395852+0000 mgr.smithi099.oodult (mgr.14180) 67 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi099:9095"}]: dispatch 2024-01-13T08:19:14.691 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:14 smithi099 conmon[33313]: cluster 2024-01-13T08:19:13.108038+0000 mgr.smithi099.oodult (mgr.14180) 68 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:14.691 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:14 smithi099 conmon[33313]: 2024-01-13T08:19:14.691 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:14 smithi099 conmon[33313]: audit 2024-01-13T08:19:13.763322+0000 mon.smithi099 (mon.0) 250 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:19:14.691 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:14 smithi099 conmon[33313]: audit 2024-01-13T08:19:13.770128+0000 mon.smithi099 (mon.0) 251 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:19:14.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:14 smithi131 conmon[38024]: cluster 2024-01-13T08:19:13.108038+0000 mgr.smithi099.oodult (mgr.14180) 68 : cluster [DBG] 2024-01-13T08:19:14.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:14 smithi131 conmon[38024]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:14.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:14 smithi131 conmon[38024]: audit 2024-01-13T08:19:13.763322+0000 mon.smithi099 (mon.0) 250 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:19:14.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:14 smithi131 conmon[38024]: audit 2024-01-13T08:19:14.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:14 smithi131 conmon[38024]: 2024-01-13T08:19:13.770128+0000 mon.smithi099 (mon.0) 251 : audit [INF] 2024-01-13T08:19:14.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:14 smithi131 conmon[38024]: from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:19:16.388 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:19:16.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:16 smithi099 conmon[33313]: cluster 2024-01-13T08:19:15.108376+0000 mgr.smithi099.oodult (mgr.14180) 69 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:16.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:16 smithi099 conmon[33313]: audit 2024-01-13T08:19:15.851231+0000 mon.smithi099 (mon.0) 252 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:19:16.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:16 smithi099 conmon[33313]: audit 2024-01-13T08:19:16.177378+0000 mon.smithi099 (mon.0) 253 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:19:16.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:16 smithi099 conmon[33313]: audit 2024-01-13T08:19:16.178283+0000 mon.smithi099 (mon.0) 254 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:19:16.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:16 smithi099 conmon[33313]: audit 2024-01-13T08:19:16.178909+0000 mon.smithi099 (mon.0) 255 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T08:19:16.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:16 smithi099 conmon[33313]: audit 2024-01-13T08:19:16.183101+0000 mon.smithi099 (mon.0) 256 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:19:16.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:16 smithi099 conmon[33313]: audit 2024-01-13T08:19:16.186550+0000 mon.smithi099 (mon.0) 257 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T08:19:16.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:16 smithi099 conmon[33313]: audit 2024-01-13T08:19:16.189263+0000 mon.smithi099 (mon.0) 258 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:19:16.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:16 smithi099 conmon[33313]: audit 2024-01-13T08:19:16.190173+0000 mon.smithi099 (mon.0) 259 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:19:16.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:16 smithi099 conmon[33313]: audit 2024-01-13T08:19:16.190431+0000 mon.smithi099 (mon.0) 260 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:19:16.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:16 smithi099 conmon[33313]: audit 2024-01-13T08:19:16.190937+0000 mon.smithi099 (mon.0) 261 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:19:16.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:16 smithi099 conmon[33313]: audit 2024-01-13T08:19:16.391932+0000 mon.smithi099 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.99:0/431692373' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:19:16.881 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:16 smithi131 conmon[38024]: cluster 2024-01-13T08:19:15.108376+0000 mgr.smithi099.oodult (mgr.14180) 69 : cluster 2024-01-13T08:19:16.882 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:16 smithi131 conmon[38024]: [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:16.882 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:16 smithi131 conmon[38024]: audit 2024-01-13T08:19:15.851231+0000 mon.smithi099 (mon.0) 252 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:19:16.882 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:16 smithi131 conmon[38024]: audit 2024-01-13T08:19:16.177378+0000 mon.smithi099 (mon.0) 253 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:19:16.882 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:16 smithi131 conmon[38024]: audit 2024-01-13T08:19:16.178283+0000 mon.smithi099 (mon.0) 254 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:19:16.882 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:16 smithi131 conmon[38024]: audit 2024-01-13T08:19:16.178909+0000 mon.smithi099 (mon.0) 255 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T08:19:16.882 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:16 smithi131 conmon[38024]: audit 2024-01-13T08:19:16.183101+0000 mon.smithi099 (mon.0) 256 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:19:16.882 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:16 smithi131 conmon[38024]: audit 2024-01-13T08:19:16.186550+0000 mon.smithi099 (mon.0) 257 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T08:19:16.882 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:16 smithi131 conmon[38024]: audit 2024-01-13T08:19:16.189263+0000 mon.smithi099 (mon.0) 258 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:19:16.882 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:16 smithi131 conmon[38024]: audit 2024-01-13T08:19:16.190173+0000 mon.smithi099 (mon.0) 259 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:19:16.882 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:16 smithi131 conmon[38024]: audit 2024-01-13T08:19:16.190431+0000 mon.smithi099 (mon.0) 260 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:19:16.882 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:16 smithi131 conmon[38024]: audit 2024-01-13T08:19:16.190937+0000 mon.smithi099 (mon.0) 261 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:19:16.882 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:16 smithi131 conmon[38024]: audit 2024-01-13T08:19:16.391932+0000 mon.smithi099 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.99:0/431692373' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:19:17.401 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":5,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2024-01-13T08:19:18.402 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:19:18.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:18 smithi099 conmon[33313]: cluster 2024-01-13T08:19:17.108798+0000 mgr.smithi099.oodult (mgr.14180) 70 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:18.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:18 smithi131 conmon[38024]: cluster 2024-01-13T08:19:17.108798+0000 mgr.smithi099.oodult (mgr.14180) 70 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:19.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:19 smithi131 conmon[38024]: audit 2024-01-13T08:19:18.831423+0000 mon.smithi131 (mon.1) 2 : audit [INF] from='client.? 172.21.15.131:0/2419382774' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "247bf7aa-d09e-44f2-8ae0-322150caee2b"}]: dispatch 2024-01-13T08:19:19.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:19 smithi131 conmon[38024]: audit 2024 2024-01-13T08:19:19.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:19 smithi131 conmon[38024]: -01-13T08:19:18.832004+0000 mon.smithi099 (mon.0) 263 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "247bf7aa-d09e-44f2-8ae0-322150caee2b"}]: dispatch 2024-01-13T08:19:19.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:19 smithi131 conmon[38024]: audit 2024-01-13T08:19:18.838110+0000 mon.smithi099 (mon.0) 264 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "247bf7aa-d09e-44f2-8ae0-322150caee2b"}]': finished 2024-01-13T08:19:19.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:19 smithi131 conmon[38024]: cluster 2024-01-13T08:19:18.838188+0000 mon.smithi099 (mon.0) 265 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-01-13T08:19:19.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:19 smithi131 conmon[38024]: audit 2024-01-13T08:19:18.838585+0000 mon.smithi099 (mon.0) 266 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:19:19.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:19 smithi131 conmon[38024]: audit 2024-01-13T08:19:19.123649+0000 mon.smithi131 (mon.1) 3 : audit [INF] from='client.? 172.21.15.131:0/3513290322' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:19:19.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:19 smithi131 conmon[38024]: audit 2024-01-13T08:19:19.124296+0000 mon.smithi099 (mon.0) 267 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:19:19.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:19 smithi131 conmon[38024]: audit 2024-01-13T08:19:19.132096+0000 mon.smithi099 (mon.0) 268 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:19:19.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:19 smithi131 conmon[38024]: cluster 2024-01-13T08:19:19.132170+0000 mon.smithi099 (mon.0) 269 : cluster [DBG] osdmap e7: 0 total, 0 up, 0 in 2024-01-13T08:19:19.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:19 smithi131 conmon[38024]: audit 2024-01-13T08:19:19.464380+0000 mon.smithi099 (mon.0) 270 : audit [INF] from='client.? 172.21.15.99:0/2775828102' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec371418-ccd8-414e-bd25-78c9ffd59f37"}]: dispatch 2024-01-13T08:19:19.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:19 smithi131 conmon[38024]: audit 2024-01-13T08:19:19.472318+0000 mon.smithi099 (mon.0) 271 : audit [INF] from='client.? 172.21.15.99:0/2775828102' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec371418-ccd8-414e-bd25-78c9ffd59f37"}]': finished 2024-01-13T08:19:19.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:19 smithi131 conmon[38024]: cluster 2024-01-13T08:19:19.472357+0000 mon.smithi099 (mon.0) 272 : cluster [DBG] osdmap e8: 1 total, 0 up, 1 in 2024-01-13T08:19:19.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:19 smithi131 conmon[38024]: audit 2024-01-13T08:19:19. 2024-01-13T08:19:19.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:19 smithi131 conmon[38024]: 472479+0000 mon.smithi099 (mon.0) 273 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:19:19.924 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:19 smithi099 conmon[33313]: audit 2024-01-13T08:19:18.831423+0000 2024-01-13T08:19:19.924 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:19 smithi099 conmon[33313]: mon.smithi131 (mon.1) 2 : audit [INF] from='client.? 172.21.15.131:0/2419382774' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "247bf7aa-d09e-44f2-8ae0-322150caee2b"}]: dispatch 2024-01-13T08:19:19.924 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:19 smithi099 conmon[33313]: audit 2024-01-13T08:19:18.832004+0000 mon.smithi099 (mon.0) 263 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "247bf7aa-d09e-44f2-8ae0-322150caee2b"}]: dispatch 2024-01-13T08:19:19.925 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:19 smithi099 conmon[33313]: audit 2024-01-13T08:19:18.838110+0000 mon.smithi099 (mon.0) 264 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "247bf7aa-d09e-44f2-8ae0-322150caee2b"}]': finished 2024-01-13T08:19:19.925 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:19 smithi099 conmon[33313]: cluster 2024-01-13T08:19:18.838188+0000 mon.smithi099 (mon.0) 265 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-01-13T08:19:19.925 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:19 smithi099 conmon[33313]: audit 2024-01-13T08:19:18.838585+0000 mon.smithi099 (mon.0) 266 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:19:19.925 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:19 smithi099 conmon[33313]: audit 2024-01-13T08:19:19.123649+0000 mon.smithi131 (mon.1) 3 : audit [INF] from='client.? 172.21.15.131:0/3513290322' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:19:19.925 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:19 smithi099 conmon[33313]: audit 2024-01-13T08:19:19.124296+0000 2024-01-13T08:19:19.925 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:19 smithi099 conmon[33313]: mon.smithi099 (mon.0) 267 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:19:19.925 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:19 smithi099 conmon[33313]: audit 2024-01-13T08:19:19.132096+0000 mon.smithi099 2024-01-13T08:19:19.925 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:19 smithi099 conmon[33313]: (mon.0) 268 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:19:19.925 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:19 smithi099 conmon[33313]: cluster 2024-01-13T08 2024-01-13T08:19:19.925 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:19 smithi099 conmon[33313]: :19:19.132170+0000 mon.smithi099 (mon. 2024-01-13T08:19:19.925 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:19 smithi099 conmon[33313]: 0) 269 : cluster [DBG] osdmap e7: 0 total, 0 up, 0 in 2024-01-13T08:19:19.925 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:19 smithi099 conmon[33313]: audit 2024 2024-01-13T08:19:19.925 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:19 smithi099 conmon[33313]: -01-13T08:19:19 2024-01-13T08:19:19.925 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:19 smithi099 conmon[33313]: .464380+0000 mon.smithi099 (mon. 2024-01-13T08:19:19.925 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:19 smithi099 conmon[33313]: 0) 270 : audit [INF] from='client.? 172.21.15.99:0/2775828102' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec371418-ccd8-414e-bd25-78c9ffd59f37"}]: dispatch 2024-01-13T08:19:19.926 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:19 smithi099 conmon[33313]: audit 2024- 2024-01-13T08:19:19.926 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:19 smithi099 conmon[33313]: 01-13T08:19:19. 2024-01-13T08:19:19.926 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:19 smithi099 conmon[33313]: 472318+0000 mon.smithi099 (mon.0) 2024-01-13T08:19:19.926 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:19 smithi099 conmon[33313]: 271 : audit [INF] from='client.? 172.21.15.99:0/2775828102' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec371418-ccd8-414e-bd25-78c9ffd59f37"}]': finished 2024-01-13T08:19:19.926 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:19 smithi099 conmon[33313]: cluster 2024-01-13T08:19:19.926 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:19 smithi099 conmon[33313]: 2024-01-13T08 2024-01-13T08:19:19.926 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:19 smithi099 conmon[33313]: :19:19.472357+0000 mon.smithi099 2024-01-13T08:19:19.926 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:19 smithi099 conmon[33313]: (mon.0) 272 : cluster 2024-01-13T08:19:19.926 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:19 smithi099 conmon[33313]: [DBG] osdmap e8: 1 total, 0 up, 1 in 2024-01-13T08:19:19.926 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:19 smithi099 conmon[33313]: audit 2024-01-13T08:19:19.926 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:19 smithi099 conmon[33313]: 2024-01-13T08:19 2024-01-13T08:19:19.926 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:19 smithi099 conmon[33313]: :19.472479+0000 mon.smithi099 ( 2024-01-13T08:19:19.926 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:19 smithi099 conmon[33313]: mon.0) 273 : audit [DBG] 2024-01-13T08:19:19.926 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:19 smithi099 conmon[33313]: from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:19:20.496 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:19:20.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:20 smithi131 conmon[38024]: cluster 2024-01-13T08:19:19.109090+0000 mgr.smithi099.oodult (mgr.14180) 71 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:20.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:20 smithi131 conmon[38024]: audit 2024-01-13T08:19:19.783934+0000 mon.smithi099 (mon.0) 274 : audit [INF] from='client.? 172.21.15.99:0/3620747549' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:19:20.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:20 smithi131 conmon[38024]: audit 2024-01-13T08:19:19.787520+0000 mon.smithi099 (mon.0) 2024-01-13T08:19:20.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:20 smithi131 conmon[38024]: 275 : audit [INF] from='client.? 172.21.15.99:0/3620747549' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:19:20.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:20 smithi131 conmon[38024]: cluster 2024 2024-01-13T08:19:20.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:20 smithi131 conmon[38024]: -01-13T08:19:19.787557+0000 mon.smithi099 (mon.0) 276 : 2024-01-13T08:19:20.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:20 smithi131 conmon[38024]: cluster [DBG] osdmap e9: 0 total, 0 up, 0 in 2024-01-13T08:19:20.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:20 smithi131 conmon[38024]: audit 2024-01-13T08:19: 2024-01-13T08:19:20.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:20 smithi131 conmon[38024]: 20.499425+0000 mon.smithi099 (mon.0) 277 : audit [DBG] 2024-01-13T08:19:20.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:20 smithi131 conmon[38024]: from='client.? 172.21.15.99:0/3698045308' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:19:20.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:20 smithi131 conmon[38024]: audit 2024-01-13T08:19:20.594620 2024-01-13T08:19:20.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:20 smithi131 conmon[38024]: +0000 mon.smithi099 (mon.0) 278 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:19:20.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:20 smithi099 conmon[33313]: cluster 2024-01-13T08:19:19.109090+0000 mgr.smithi099.oodult (mgr.14180) 71 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:20.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:20 smithi099 conmon[33313]: 2024-01-13T08:19:20.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:20 smithi099 conmon[33313]: audit 2024-01-13T08:19:19.783934+0000 mon.smithi099 (mon.0) 274 : audit [INF] from='client.? 172.21.15.99:0/3620747549' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:19:20.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:20 smithi099 conmon[33313]: audit 2024-01-13T08:19:19.787520+0000 mon.smithi099 (mon.0) 275 : audit [INF] from='client.? 172.21.15.99:0/3620747549' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:19:20.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:20 smithi099 conmon[33313]: cluster 2024-01-13T08:19:19.787557+0000 mon.smithi099 (mon.0) 276 : cluster 2024-01-13T08:19:20.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:20 smithi099 conmon[33313]: [DBG] osdmap e9: 0 total, 0 up, 0 in 2024-01-13T08:19:20.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:20 smithi099 conmon[33313]: audit 2024-01-13T08:19:20.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:20 smithi099 conmon[33313]: 2024-01-13T08:19:20.499425+0000 mon.smithi099 (mon.0) 2024-01-13T08:19:20.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:20 smithi099 conmon[33313]: 277 : audit [DBG] from='client.? 172.21.15.99:0/3698045308' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:19:20.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:20 smithi099 conmon[33313]: audit 2024-01-13T08:19 2024-01-13T08:19:20.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:20 smithi099 conmon[33313]: :20.594620+0000 mon.smithi099 (mon.0) 278 : audit [DBG] 2024-01-13T08:19:20.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:20 smithi099 conmon[33313]: from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:19:21.057 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":9,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705133958,"num_remapped_pgs":0} 2024-01-13T08:19:21.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: cephadm 2024-01-13T08:19:19.798975+0000 mgr.smithi099.oodult (mgr.14180) 72 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi131', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:19:21.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:19:21.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:19:21.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: return f(*arg) 2024-01-13T08:19:21.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:19:21.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:19:21.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:19:21.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: code, '\n'.join(err))) 2024-01-13T08:19:21.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:19:21.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp2kiohah5:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpn3fw54zo:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:19:21.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:19:21.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:19:21.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:19:21.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 247bf7aa-d09e-44f2-8ae0-322150caee2b 2024-01-13T08:19:21.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:19:21.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:19:21.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:19:21.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:19:21.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:19:21.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr self.prepare() 2024-01-13T08:19:21.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:21.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:21.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:19:21.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:19:21.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:19:21.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:19:21.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:19:21.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:19:21.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:19:21.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:19:21.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:19:21.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:19:21.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:19:21.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:19:21.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:19:21.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:19:21.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:19:21.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:19:21.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:19:21.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:19:21.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:19:21.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:19:21.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:19:21.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:19:21.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:19:21.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:19:21.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:19:21.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:21.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:21.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:19:21.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:19:21.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:19:21.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:19:21.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:21.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:21.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:19:21.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:19:21.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:19:21.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:19:21.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:19:21.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:19:21.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:19:21.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:19:21.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:21.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:21.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:19:21.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:19:21.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:19:21.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:19:21.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:19:21.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:19:21.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:19:21.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: main() 2024-01-13T08:19:21.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:19:21.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: r = ctx.func(ctx) 2024-01-13T08:19:21.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:19:21.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:19:21.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:19:21.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:19:21.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:19:21.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:19:21.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774 2024-01-13T08:19:21.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:19:21.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:19:21.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:19:21.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:19:21.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:19:21.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:19:21.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp2kiohah5:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpn3fw54zo:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:19:21.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: cephadm 2024-01-13T08:19:20.591689+0000 mgr.smithi099.oodult (mgr.14180) 73 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi131', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:19:21.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:19:21.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:19:21.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: return f(*arg) 2024-01-13T08:19:21.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:19:21.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:19:21.907 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:19:21.907 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: code, '\n'.join(err))) 2024-01-13T08:19:21.907 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi099/config 2024-01-13T08:19:21.907 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpj6ewhgwq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmprge5a63v:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:19:21.907 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:19:21.907 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:19:21.907 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:19:21.907 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new ec371418-ccd8-414e-bd25-78c9ffd59f37 2024-01-13T08:19:21.907 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:19:21.907 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:19:21.907 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:19:21.907 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:19:21.907 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:19:21.907 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr self.prepare() 2024-01-13T08:19:21.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:21.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:21.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:19:21.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:19:21.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:19:21.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:19:21.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:19:21.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:19:21.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:19:21.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:19:21.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:19:21.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:19:21.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:19:21.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:19:21.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:19:21.909 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:19:21.909 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:19:21.909 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:19:21.909 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:19:21.909 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:19:21.909 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:19:21.909 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:19:21.909 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:19:21.910 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:19:21.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:19:21.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:19:21.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:19:21.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:21.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:21.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:19:21.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:19:21.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:19:21.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:19:21.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:21.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:21.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:19:21.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:19:21.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:19:21.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:19:21.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_ 2024-01-13T08:19:21.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: osd 2024-01-13T08:19:21.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:19:21.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:19:21.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:19:21.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:21.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:21.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:19:21.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:19:21.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:19:21.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:19:21.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:19:21.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:19:21.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:19:21.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: main() 2024-01-13T08:19:21.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:19:21.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: r = ctx.func(ctx) 2024-01-13T08:19:21.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:19:21.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:19:21.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:19:21.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:19:21.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:19:21.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:19:21.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:19:21.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:19:21.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:19:21.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:19:21.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:19:21.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:19:21.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpj6ewhgwq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmprge5a63v:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:19:21.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: cephadm 2024-01-13T08:19:20.593349+0000 mgr.smithi099.oodult (mgr.14180) 74 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T08:19:21.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: service_id: all-available-devices 2024-01-13T08:19:21.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: service_name: osd.all-available-devices 2024-01-13T08:19:21.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: placement: 2024-01-13T08:19:21.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: host_pattern: '*' 2024-01-13T08:19:21.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: spec: 2024-01-13T08:19:21.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: data_devices: 2024-01-13T08:19:21.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: all: true 2024-01-13T08:19:21.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: filter_logic: AND 2024-01-13T08:19:21.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: objectstore: bluestore 2024-01-13T08:19:21.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:19:21.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp2kiohah5:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpn3fw54zo:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:19:21.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:19:21.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:19:21.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:19:21.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 247bf7aa-d09e-44f2-8ae0-322150caee2b 2024-01-13T08:19:21.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:19:21.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:19:21.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:19:21.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:19:21.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:19:21.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr self.prepare() 2024-01-13T08:19:21.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:21.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:21.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:19:21.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:19:21.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:19:21.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:19:21.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:19:21.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:19:21.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:19:21.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:19:21.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:19:21.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:19:21.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:19:21.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:19:21.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:19:21.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:19:21.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:19:21.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:19:21.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:19:21.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:19:21.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:19:21.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:19:21.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:19:21.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main. 2024-01-13T08:19:21.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: py", line 46, in main 2024-01-13T08:19:21.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:19:21.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:19:21.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:19:21.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:21.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:21.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:19:21.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:19:21.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:19:21.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:19:21.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:21.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:21.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:19:21.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:19:21.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:19:21.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:19:21.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:19:21.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:19:21.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:19:21.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:19:21.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:21.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:21.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:19:21.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:19:21.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:19:21.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:19:21.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:19:21.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:19:21.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:19:21.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: main() 2024-01-13T08:19:21.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:19:21.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: r = ctx.func(ctx) 2024-01-13T08:19:21.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:19:21.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:19:21.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:19:21.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:19:21.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:19:21.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:19:21.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:19:21.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:19:21.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:19:21.920 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:19:21.920 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:19:21.920 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:19:21.920 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp2kiohah5:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpn3fw54zo:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:19:21.920 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:19:21.920 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T08:19:21.920 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: if self._apply_service(spec): 2024-01-13T08:19:21.920 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T08:19:21.920 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T08:19:21.920 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T08:19:21.920 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T08:19:21.920 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T08:19:21.920 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T08:19:21.920 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T08:19:21.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T08:19:21.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T08:19:21.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: raise self._value 2024-01-13T08:19:21.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T08:19:21.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: result = (True, func(*args, **kwds)) 2024-01-13T08:19:21.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T08:19:21.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: return list(map(*args)) 2024-01-13T08:19:21.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:19:21.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: return f(*arg) 2024-01-13T08:19:21.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:19:21.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:19:21.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:19:21.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: code, '\n'.join(err))) 2024-01-13T08:19:21.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:19:21.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp2kiohah5:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpn3fw54zo:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:19:21.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:19:21.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:19:21.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:19:21.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 247bf7aa-d09e-44f2-8ae0-322150caee2b 2024-01-13T08:19:21.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr --> Was unable 2024-01-13T08:19:21.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: to complete a new OSD, will rollback changes 2024-01-13T08:19:21.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:19:21.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:19:21.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:19:21.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:19:21.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr self.prepare() 2024-01-13T08:19:21.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:21.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:21.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:19:21.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:19:21.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:19:21.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:19:21.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:19:21.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:19:21.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:19:21.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:19:21.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:19:21.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:19:21.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:19:21.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:19:21.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:19:21.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:19:21.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:19:21.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:19:21.924 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:19:21.924 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:19:21.924 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:19:21.924 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:19:21.924 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:19:21.924 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:19:21.924 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:19:21.924 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:19:21.924 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:19:21.924 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:21.924 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:21.924 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:19:21.924 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:19:21.924 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:19:21.924 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:19:21.925 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:21.925 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:21.925 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:19:21.925 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:19:21.925 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:19:21.925 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:19:21.925 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:19:21.925 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:19:21.925 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:19:21.925 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:19:21.925 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:21.925 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:21.925 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:19:21.925 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:19:21.926 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:19:21.926 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:19:21.926 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:19:21.926 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:19:21.926 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:19:21.926 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: main() 2024-01-13T08:19:21.926 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:19:21.926 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: r = ctx.func(ctx) 2024-01-13T08:19:21.926 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:19:21.926 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:19:21.926 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:19:21.926 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:19:21.926 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:19:21.926 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:19:21.927 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:19:21.927 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:19:21.927 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:19:21.927 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:19:21.927 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:19:21.927 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:19:21.927 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp2kiohah5:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpn3fw54zo:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:19:21.927 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: cluster 2024-01-13T08:19:20.593891+0000 mgr.smithi099.oodult (mgr.14180) 75 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:21.927 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:21 smithi131 conmon[38024]: cluster 2024-01-13T08:19:20.837142+0000 mon.smithi099 (mon.0) 279 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T08:19:21.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: cephadm 2024-01-13T08:19:19.798975+0000 mgr.smithi099.oodult 2024-01-13T08:19:21.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: (mgr.14180) 72 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi131', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:19:21.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:19:21.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:19:21.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: return f(*arg) 2024-01-13T08:19:21.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:19:21.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:19:21.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:19:21.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: code, '\n'.join(err))) 2024-01-13T08:19:21.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:19:21.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp2kiohah5:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpn3fw54zo:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:19:21.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:19:21.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:19:21.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:19:21.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 247bf7aa-d09e-44f2-8ae0-322150caee2b 2024-01-13T08:19:21.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:19:21.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:19:21.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:19:21.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:19:21.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:19:21.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr self.prepare() 2024-01-13T08:19:21.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:21.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:21.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:19:21.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:19:21.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:19:21.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:19:21.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:19:21.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:19:21.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:19:21.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:19:21.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:19:21.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:19:21.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:19:21.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:19:21.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:19:21.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:19:21.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:19:21.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:19:21.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:19:21.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:19:21.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:19:21.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:19:21.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:19:21.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:19:21.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:19:21.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:19:21.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:19:21.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:21.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:21.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:19:21.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:19:21.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:19:21.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:19:21.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:21.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:21.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:19:21.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:19:21.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:19:21.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:19:21.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:19:21.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:19:21.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:19:21.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:19:21.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:21.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:21.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:19:21.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:19:21.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:19:21.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:19:21.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:19:21.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:19:21.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:19:21.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: main() 2024-01-13T08:19:21.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:19:21.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: r = ctx.func(ctx) 2024-01-13T08:19:21.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:19:21.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:19:21.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:19:21.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:19:21.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:19:21.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:19:21.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39f 2024-01-13T08:19:21.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: ebbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:19:21.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:19:21.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:19:21.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:19:21.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:19:21.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:19:21.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp2kiohah5:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpn3fw54zo:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:19:21.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: cephadm 2024-01-13T08:19:20.591689+0000 mgr.smithi099.oodult (mgr.14180) 73 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi131', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:19:21.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:19:21.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:19:21.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: return f(*arg) 2024-01-13T08:19:21.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:19:21.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:19:21.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:19:21.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: code, '\n'.join(err))) 2024-01-13T08:19:21.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi099/config 2024-01-13T08:19:21.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpj6ewhgwq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmprge5a63v:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:19:21.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:19:21.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:19:21.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:19:21.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new ec371418-ccd8-414e-bd25-78c9ffd59f37 2024-01-13T08:19:21.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:19:21.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:19:21.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:19:21.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:19:21.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:19:21.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr self.prepare() 2024-01-13T08:19:21.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:21.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:21.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:19:21.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:19:21.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:19:21.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:19:21.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:19:21.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:19:21.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:19:21.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:19:21.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:19:21.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:19:21.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:19:21.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:19:21.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:19:21.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:19:21.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:19:21.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:19:21.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:19:21.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:19:21.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:19:21.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:19:21.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:19:21.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:19:21.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:19:21.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:19:21.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:19:21.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:21.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:21.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:19:21.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:19:21.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:19:21.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:19:21.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:21.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:21.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:19:21.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:19:21.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:19:21.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:19:21.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:19:21.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr Zap(['--destroy', '--osd-id', os 2024-01-13T08:19:21.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: d_id]).main() 2024-01-13T08:19:21.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:19:21.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:19:21.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:21.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:21.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:19:21.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:19:21.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:19:21.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:19:21.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:19:21.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:19:21.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:19:21.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: main() 2024-01-13T08:19:21.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:19:21.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: r = ctx.func(ctx) 2024-01-13T08:19:21.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:19:21.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:19:21.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:19:21.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:19:21.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:19:21.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:19:21.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:19:21.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:19:21.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:19:21.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:19:21.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:19:21.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:19:21.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpj6ewhgwq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmprge5a63v:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:19:21.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: cephadm 2024-01-13T08:19:20.593349+0000 mgr.smithi099.oodult (mgr.14180) 74 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T08:19:21.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: service_id: all-available-devices 2024-01-13T08:19:21.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: service_name: osd.all-available-devices 2024-01-13T08:19:21.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: placement: 2024-01-13T08:19:21.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: host_pattern: '*' 2024-01-13T08:19:21.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: spec: 2024-01-13T08:19:21.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: data_devices: 2024-01-13T08:19:21.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: all: true 2024-01-13T08:19:21.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: filter_logic: AND 2024-01-13T08:19:21.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: objectstore: bluestore 2024-01-13T08:19:21.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:19:21.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp2kiohah5:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpn3fw54zo:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:19:21.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:19:21.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:19:21.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:19:21.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 247bf7aa-d09e-44f2-8ae0-322150caee2b 2024-01-13T08:19:21.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:19:21.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:19:21.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:19:21.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:19:21.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:19:21.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr self.prepare() 2024-01-13T08:19:21.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:21.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:21.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:19:21.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:19:21.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:19:21.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:19:21.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:19:21.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:19:21.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:19:21.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:19:21.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:19:21.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:19:21.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:19:21.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:19:21.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:19:21.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:19:21.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:19:21.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:19:21.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:19:21.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:19:21.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:19:21.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:19:21.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:19:21.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:19:21.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispa 2024-01-13T08:19:21.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: tch(self.mapper, self.argv) 2024-01-13T08:19:21.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:19:21.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:19:21.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:21.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:21.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:19:21.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:19:21.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:19:22.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:19:22.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:22.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:22.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:19:22.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:19:22.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:19:22.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:19:22.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:19:22.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:19:22.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:19:22.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:19:22.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:22.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:22.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:19:22.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:19:22.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:19:22.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:19:22.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:19:22.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:19:22.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:19:22.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: main() 2024-01-13T08:19:22.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:19:22.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: r = ctx.func(ctx) 2024-01-13T08:19:22.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:19:22.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:19:22.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:19:22.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:19:22.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:19:22.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:19:22.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:19:22.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:19:22.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:19:22.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:19:22.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:19:22.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:19:22.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp2kiohah5:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpn3fw54zo:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:19:22.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:19:22.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T08:19:22.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: if self._apply_service(spec): 2024-01-13T08:19:22.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T08:19:22.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T08:19:22.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T08:19:22.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T08:19:22.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T08:19:22.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T08:19:22.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T08:19:22.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T08:19:22.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T08:19:22.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: raise self._value 2024-01-13T08:19:22.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T08:19:22.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: result = (True, func(*args, **kwds)) 2024-01-13T08:19:22.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T08:19:22.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: return list(map(*args)) 2024-01-13T08:19:22.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:19:22.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: return f(*arg) 2024-01-13T08:19:22.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:19:22.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:19:22.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:19:22.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: code, '\n'.join(err))) 2024-01-13T08:19:22.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:19:22.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp2kiohah5:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpn3fw54zo:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:19:22.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:19:22.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:19:22.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:19:22.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 247bf7aa-d09e-44f2-8ae0-322150caee2b 2024-01-13T08:19:22.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:19:22.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: st 2024-01-13T08:19:22.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: derr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:19:22.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:19:22.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:19:22.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:19:22.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr self.prepare() 2024-01-13T08:19:22.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:22.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:22.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:19:22.005 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:19:22.005 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:19:22.005 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:19:22.005 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:19:22.005 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:19:22.005 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:19:22.005 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:19:22.005 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:19:22.005 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:19:22.005 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:19:22.005 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:19:22.005 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:19:22.005 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:19:22.005 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:19:22.005 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:19:22.006 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:19:22.006 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:19:22.006 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:19:22.006 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:19:22.006 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:19:22.006 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:19:22.006 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:19:22.006 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:19:22.006 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:19:22.006 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:22.006 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:22.006 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:19:22.006 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:19:22.006 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:19:22.006 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:19:22.007 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:22.007 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:22.007 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:19:22.007 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:19:22.007 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:19:22.007 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:19:22.007 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:19:22.007 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:19:22.007 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:19:22.007 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:19:22.007 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:22.007 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:22.007 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:19:22.007 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:19:22.007 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:19:22.007 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:19:22.008 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:19:22.008 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:19:22.008 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:19:22.008 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: main() 2024-01-13T08:19:22.008 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:19:22.008 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: r = ctx.func(ctx) 2024-01-13T08:19:22.008 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:19:22.008 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:19:22.008 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:19:22.008 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:19:22.008 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:19:22.008 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:19:22.008 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:19:22.008 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:19:22.009 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:19:22.009 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:19:22.009 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:19:22.009 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:19:22.009 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp2kiohah5:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpn3fw54zo:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:19:22.009 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: cluster 2024-01-13T08:19:20.593891+0000 mgr.smithi099.oodult (mgr.14180) 75 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:22.009 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:21 smithi099 conmon[33313]: cluster 2024-01-13T08:19:20.837142+0000 mon.smithi099 (mon.0) 279 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T08:19:22.058 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:19:23.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:22 smithi099 conmon[33313]: audit 2024-01-13T08:19:21.979299+0000 mon.smithi099 (mon.0) 280 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:19:23.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:22 smithi099 conmon[33313]: audit 2024-01-13T08:19:22.244774+0000 mon.smithi099 (mon.0) 281 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:19:23.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:22 smithi099 conmon[33313]: cluster 2024-01-13T08:19:22.594260+0000 mgr.smithi099.oodult (mgr.14180) 76 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:23.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:22 smithi131 conmon[38024]: audit 2024-01-13T08:19:21.979299+0000 mon.smithi099 (mon.0) 280 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:19:23.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:22 smithi131 conmon[38024]: audit 2024-01-13T08:19:22.244774+0000 mon.smithi099 (mon.0) 281 : audit 2024-01-13T08:19:23.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:22 smithi131 conmon[38024]: [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:19:23.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:22 smithi131 conmon[38024]: cluster 2024-01-13T08:19:22.594260+0000 mgr.smithi099.oodult (mgr.14180) 76 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:24.266 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:19:24.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:24 smithi099 conmon[33313]: audit 2024-01-13T08:19:24.269794+0000 mon.smithi099 (mon. 2024-01-13T08:19:24.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:24 smithi099 conmon[33313]: 0) 282 : audit [DBG] from='client.? 172.21.15.99:0/812436860' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:19:24.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:24 smithi131 conmon[38024]: audit 2024-01-13T08:19:24.269794+0000 mon.smithi099 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.99:0/812436860' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:19:24.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:24 smithi131 conmon[38024]: 2024-01-13T08:19:25.029 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":9,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705133958,"num_remapped_pgs":0} 2024-01-13T08:19:25.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:25 smithi131 conmon[38024]: audit 2024-01-13T08:19:24.411003+0000 mon.smithi099 (mon.0) 283 : audit 2024-01-13T08:19:25.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:25 smithi131 conmon[38024]: [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:19:25.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:25 smithi131 conmon[38024]: audit 2024-01-13T08:19:24.411686+0000 mon.smithi099 (mon.0) 284 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:19:25.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:25 smithi131 conmon[38024]: audit 2024-01-13T08:19:24.412287+0000 mon.smithi099 (mon.0) 285 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T08:19:25.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:25 smithi131 conmon[38024]: audit 2024-01-13T08:19:24.416226+0000 mon.smithi099 (mon.0) 286 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:19:25.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:25 smithi131 conmon[38024]: cluster 2024-01-13T08:19:24.416546+0000 mgr.smithi099.oodult (mgr.14180) 77 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:25.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:25 smithi131 conmon[38024]: audit 2024-01-13T08:19:24.419516+0000 mon.smithi099 (mon.0) 287 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T08:19:25.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:25 smithi131 conmon[38024]: audit 2024-01-13T08:19:24.422111+0000 mon.smithi099 (mon.0) 288 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:19:25.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:25 smithi131 conmon[38024]: audit 2024- 2024-01-13T08:19:25.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:25 smithi131 conmon[38024]: 01-13T08:19:24.422908+0000 mon.smithi099 (mon.0) 289 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:19:25.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:25 smithi131 conmon[38024]: audit 2024-01-13 2024-01-13T08:19:25.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:25 smithi131 conmon[38024]: T08:19:24.423115+0000 mon.smithi099 (mon.0 2024-01-13T08:19:25.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:25 smithi131 conmon[38024]: ) 290 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:19:25.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:25 smithi131 conmon[38024]: audit 2024-01-13T08:19:25.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:25 smithi131 conmon[38024]: 2024-01-13T08:19: 2024-01-13T08:19:25.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:25 smithi131 conmon[38024]: 24.423593+0000 mon.smithi099 (mon.0) 2024-01-13T08:19:25.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:25 smithi131 conmon[38024]: 291 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:19:25.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:25 smithi099 conmon[33313]: audit 2024-01-13T08:19:24.411003+0000 mon.smithi099 (mon.0) 283 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:19:25.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:25 smithi099 conmon[33313]: audit 2024-01-13T08:19:24.411686+0000 mon.smithi099 (mon.0) 284 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:19:25.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:25 smithi099 conmon[33313]: audit 2024-01-13T08:19:24.412287+0000 mon.smithi099 (mon.0) 285 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T08:19:25.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:25 smithi099 conmon[33313]: audit 2024-01-13T08:19:24.416226+0000 mon.smithi099 (mon.0) 286 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:19:25.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:25 smithi099 conmon[33313]: cluster 2024-01-13T08:19:24.416546+0000 mgr.smithi099.oodult (mgr.14180) 77 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:25.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:25 smithi099 conmon[33313]: audit 2024-01-13T08:19:24.419516+0000 mon.smithi099 (mon.0) 287 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T08:19:25.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:25 smithi099 conmon[33313]: audit 2024-01-13T08:19:24.422111+0000 mon.smithi099 (mon.0) 288 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:19:25.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:25 smithi099 conmon[33313]: audit 2024-01-13 2024-01-13T08:19:25.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:25 smithi099 conmon[33313]: T08:19:24.422908+0000 mon.smithi099 (mon.0) 289 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:19:25.658 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:25 smithi099 conmon[33313]: audit 2024-01-13T08:19:24.423115+0000 mon.smithi099 (mon.0) 290 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:19:25.659 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:25 smithi099 conmon[33313]: audit 2024-01-13T08:19:24.423593+0000 mon.smithi099 (mon.0) 291 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:19:26.031 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:19:26.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:26 smithi099 conmon[33313]: cluster 2024-01-13T08:19:25.414835 2024-01-13T08:19:26.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:26 smithi099 conmon[33313]: +0000 mon.smithi099 (mon.0) 292 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T08:19:26.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:26 smithi099 conmon[33313]: cluster 2024-01-13T08:19:25.414855+0000 mon.smithi099 (mon.0) 293 : cluster [INF] Cluster is now healthy 2024-01-13T08:19:26.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:26 smithi131 conmon[38024]: cluster 2024-01-13T08:19:25.414835+0000 mon.smithi099 (mon.0) 292 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T08:19:26.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:26 smithi131 conmon[38024]: cluster 2024-01-13T08:19:25.414855+0000 mon.smithi099 (mon.0) 293 : cluster [INF] Cluster is now healthy 2024-01-13T08:19:27.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:27 smithi131 conmon[38024]: cluster 2024-01-13T08:19:26.416738+0000 mgr.smithi099.oodult (mgr.14180) 78 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:27.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:27 smithi131 conmon[38024]: audit 2024-01-13T08:19:26.929058+0000 mon.smithi131 (mon.1) 4 : audit [INF] from='client.? 172.21.15.131:0/3737991811' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5600c88-a4b1-4f42-a8c4-6b1306b1cdeb"}]: dispatch 2024-01-13T08:19:27.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:27 smithi131 conmon[38024]: audit 2024-01-13T08:19:26.929756+0000 mon.smithi099 (mon.0) 294 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5600c88-a4b1-4f42-a8c4-6b1306b1cdeb"}]: dispatch 2024-01-13T08:19:27.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:27 smithi131 conmon[38024]: audit 2024-01-13T08:19:26.933797+0000 mon.smithi099 (mon.0) 295 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c5600c88-a4b1-4f42-a8c4-6b1306b1cdeb"}]': finished 2024-01-13T08:19:27.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:27 smithi131 conmon[38024]: cluster 2024-01-13T08:19:26.933867 2024-01-13T08:19:27.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:27 smithi131 conmon[38024]: +0000 mon.smithi099 (mon.0) 296 : cluster [DBG] osdmap e10: 1 total, 0 up, 1 in 2024-01-13T08:19:27.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:27 smithi131 conmon[38024]: audit 2024-01-13T08:19:26.933967+0000 2024-01-13T08:19:27.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:27 smithi131 conmon[38024]: mon.smithi099 (mon.0) 297 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:19:27.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:27 smithi131 conmon[38024]: audit 2024-01-13T08:19:27.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:27 smithi131 conmon[38024]: 2024-01-13T08:19:27.220957+0000 mon.smithi131 2024-01-13T08:19:27.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:27 smithi131 conmon[38024]: (mon.1) 5 : audit [INF] from='client.? 172.21.15.131:0/2289393826' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:19:27.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:27 smithi131 conmon[38024]: audit 2024-01-13T08 2024-01-13T08:19:27.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:27 smithi131 conmon[38024]: :19:27.221419+0000 mon.smithi099 (mon.0) 298 : audit [INF] 2024-01-13T08:19:27.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:27 smithi131 conmon[38024]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:19:27.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:27 smithi131 conmon[38024]: audit 2024 2024-01-13T08:19:27.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:27 smithi131 conmon[38024]: -01-13T08:19:27.225568+0000 mon.smithi099 (mon.0) 299 : audit [INF] 2024-01-13T08:19:27.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:27 smithi131 conmon[38024]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:19:27.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:27 smithi131 conmon[38024]: cluster 2024 2024-01-13T08:19:27.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:27 smithi131 conmon[38024]: -01-13T08:19:27.225639+0000 mon.smithi099 (mon.0) 300 : cluster 2024-01-13T08:19:27.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:27 smithi131 conmon[38024]: [DBG] osdmap e11: 0 total, 0 up, 0 in 2024-01-13T08:19:27.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:27 smithi131 conmon[38024]: audit 2024- 2024-01-13T08:19:27.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:27 smithi131 conmon[38024]: 01-13T08:19:27.382328+0000 mon.smithi099 (mon.0) 301 : audit [INF] from='client.? 172.21.15.99:0/1449148183' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ef239a7-91e1-4ac5-a0a3-aa7e23b0f567"}]: dispatch 2024-01-13T08:19:27.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:27 smithi131 conmon[38024]: 2024-01-13T08:19:27.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:27 smithi131 conmon[38024]: audit 2024-01 2024-01-13T08:19:27.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:27 smithi131 conmon[38024]: -13T08:19:27.386479+0000 mon.smithi099 (mon.0) 302 2024-01-13T08:19:27.652 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:27 smithi131 conmon[38024]: : audit [INF] from='client.? 172.21.15.99:0/1449148183' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2ef239a7-91e1-4ac5-a0a3-aa7e23b0f567"}]': finished 2024-01-13T08:19:27.652 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:27 smithi131 conmon[38024]: cluster 2024-01-13T08:19:27.652 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:27 smithi131 conmon[38024]: 2024-01-13T08:19:27.386524+0000 2024-01-13T08:19:27.652 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:27 smithi131 conmon[38024]: mon.smithi099 (mon.0) 303 : cluster [DBG] osdmap e12: 1 total, 0 up, 1 in 2024-01-13T08:19:27.652 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:27 smithi131 conmon[38024]: audit 2024-01-13T08:19:27.652 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:27 smithi131 conmon[38024]: 2024-01-13T08:19:27.386646+0000 mon.smithi099 (mon 2024-01-13T08:19:27.652 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:27 smithi131 conmon[38024]: .0) 304 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:19:27.685 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:27 smithi099 conmon[33313]: cluster 2024-01-13T08:19:26.416738+0000 mgr.smithi099.oodult (mgr.14180) 78 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:27.685 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:27 smithi099 conmon[33313]: audit 2024-01-13T08:19:26.929058+0000 mon.smithi131 (mon.1) 4 : audit [INF] from='client.? 172.21.15.131:0/3737991811' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5600c88-a4b1-4f42-a8c4-6b1306b1cdeb"}]: dispatch 2024-01-13T08:19:27.685 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:27 smithi099 conmon[33313]: audit 2024-01-13T08:19:26.929756+0000 mon.smithi099 (mon.0) 294 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5600c88-a4b1-4f42-a8c4-6b1306b1cdeb"}]: dispatch 2024-01-13T08:19:27.686 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:27 smithi099 conmon[33313]: audit 2024-01-13T08:19:26.933797+0000 mon.smithi099 (mon.0) 295 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c5600c88-a4b1-4f42-a8c4-6b1306b1cdeb"}]': finished 2024-01-13T08:19:27.686 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:27 smithi099 conmon[33313]: cluster 2024-01-13T08:19:26.933867+0000 mon.smithi099 (mon.0) 296 : cluster [DBG] osdmap e10: 1 total, 0 up, 1 in 2024-01-13T08:19:27.686 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:27 smithi099 conmon[33313]: audit 2024-01-13T08:19:26.933967+0000 mon.smithi099 (mon.0) 297 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:19:27.686 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:27 smithi099 conmon[33313]: audit 2024-01-13T08:19:27.220957+0000 mon.smithi131 (mon.1) 5 : audit [INF] from='client.? 172.21.15.131:0/2289393826' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:19:27.686 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:27 smithi099 conmon[33313]: audit 2024-01-13T08:19:27.221419+0000 mon.smithi099 (mon.0) 298 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:19:27.686 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:27 smithi099 conmon[33313]: audit 2024-01-13 2024-01-13T08:19:27.686 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:27 smithi099 conmon[33313]: T08:19:27.225568+0000 mon.smithi099 (mon.0) 299 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:19:27.686 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:27 smithi099 conmon[33313]: cluster 2024-01-13T08:19:27.225639+0000 mon.smithi099 (mon.0) 300 : cluster [DBG] osdmap e11: 0 total, 0 up, 0 in 2024-01-13T08:19:27.686 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:27 smithi099 conmon[33313]: audit 2024-01-13T08:19:27.382328+0000 mon.smithi099 (mon.0) 301 : audit [INF] from='client.? 172.21.15.99:0/1449148183' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ef239a7-91e1-4ac5-a0a3-aa7e23b0f567"}]: dispatch 2024-01-13T08:19:27.687 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:27 smithi099 conmon[33313]: audit 2024-01-13T08:19:27.386479+0000 mon.smithi099 (mon.0) 302 : audit [INF] from='client.? 172.21.15.99:0/1449148183' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2ef239a7-91e1-4ac5-a0a3-aa7e23b0f567"}]': finished 2024-01-13T08:19:27.687 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:27 smithi099 conmon[33313]: cluster 2024-01-13T08:19:27.386524+0000 mon.smithi099 (mon.0) 303 : cluster [DBG] osdmap e12: 1 total, 0 up, 1 in 2024-01-13T08:19:27.687 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:27 smithi099 conmon[33313]: audit 2024-01-13T08:19:27.386646+0000 mon.smithi099 (mon.0) 304 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:19:28.058 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:19:28.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:28 smithi099 conmon[33313]: audit 2024-01-13T08:19:27.699224+0000 mon.smithi099 (mon.0) 305 : audit [INF] 2024-01-13T08:19:28.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:28 smithi099 conmon[33313]: from='client.? 172.21.15.99:0/2608050314' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:19:28.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:28 smithi099 conmon[33313]: audit 2024-01-13T08:19:27.703242+0000 mon.smithi099 (mon.0) 306 : audit [INF] from='client.? 172.21.15.99:0/2608050314' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:19:28.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:28 smithi099 conmon[33313]: cluster 2024-01-13T 2024-01-13T08:19:28.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:28 smithi099 conmon[33313]: 08:19:27.703291+0000 mon.smithi099 (mon.0) 307 : cluster [DBG] osdmap e13: 0 total, 0 up, 0 in 2024-01-13T08:19:28.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:28 smithi099 conmon[33313]: audit 2024- 2024-01-13T08:19:28.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:28 smithi099 conmon[33313]: 01-13T08:19:28.061259+0000 mon.smithi099 (mon.0) 308 2024-01-13T08:19:28.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:28 smithi099 conmon[33313]: : audit [DBG] from='client.? 172.21.15.99:0/3034098187' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:19:28.851 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705133958,"num_remapped_pgs":0} 2024-01-13T08:19:28.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:28 smithi131 conmon[38024]: audit 2024-01-13T08:19:27.699224+0000 mon.smithi099 (mon.0) 305 : audit [INF] 2024-01-13T08:19:28.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:28 smithi131 conmon[38024]: from='client.? 172.21.15.99:0/2608050314' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:19:28.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:28 smithi131 conmon[38024]: audit 2024-01-13T08:19:27.703242+0000 mon.smithi099 (mon.0) 306 : audit [INF] from='client.? 172.21.15.99:0/2608050314' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:19:28.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:28 smithi131 conmon[38024]: cluster 2024-01-13T08:19:27.703291+0000 mon.smithi099 (mon.0) 307 2024-01-13T08:19:28.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:28 smithi131 conmon[38024]: : cluster [DBG] osdmap e13: 0 total, 0 up, 0 in 2024-01-13T08:19:28.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:28 smithi131 conmon[38024]: audit 2024-01-13T08:19:28.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:28 smithi131 conmon[38024]: 2024-01-13T08:19:28.061259+0000 mon.smithi099 (mon.0) 308 : audit 2024-01-13T08:19:28.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:28 smithi131 conmon[38024]: [DBG] from='client.? 172.21.15.99:0/3034098187' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:19:29.852 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:19:29.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: cephadm 2024-01-13T08:19:27.987431+0000 mgr.smithi099.oodult (mgr.14180) 79 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi131', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:19:29.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:19:29.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:19:29.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: return f(*arg) 2024-01-13T08:19:29.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:19:29.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:19:29.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:19:29.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: code, '\n'.join(err))) 2024-01-13T08:19:29.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:19:29.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpqu5auolp:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp6ugz_hqr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:19:29.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:19:29.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:19:29.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:19:29.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new c5600c88-a4b1-4f42-a8c4-6b1306b1cdeb 2024-01-13T08:19:29.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:19:29.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:19:29.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:19:29.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:19:29.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:19:29.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr self.prepare() 2024-01-13T08:19:29.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:29.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:29.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:19:29.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:19:29.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:19:29.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:19:29.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:19:29.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:19:29.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:19:29.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:19:29.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:19:29.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:19:29.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:19:29.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:19:29.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:19:29.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:19:29.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:19:29.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:19:29.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:19:29.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:19:29.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:19:29.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:19:29.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:19:29.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:19:29.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:19:29.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:19:29.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:19:29.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:29.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:29.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:19:29.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:19:29.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:19:29.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:19:29.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:29.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:29.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:19:29.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:19:29.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:19:29.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:19:29.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:19:29.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:19:29.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:19:29.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:19:29.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:29.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:29.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:19:29.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:19:29.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:19:29.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:19:29.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:19:29.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:19:29.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:19:29.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: main() 2024-01-13T08:19:29.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:19:29.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: r = ctx.func(ctx) 2024-01-13T08:19:29.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:19:29.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:19:29.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:19:29.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:19:29.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:19:29.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:19:29.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774 2024-01-13T08:19:29.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:19:29.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:19:29.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:19:29.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:19:29.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:19:29.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:19:29.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpqu5auolp:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp6ugz_hqr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:19:29.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:29 smithi099 conmon[33313]: cluster 2024-01-13T08:19:28.417060+0000 mgr.smithi099.oodult (mgr.14180) 80 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:30.150 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: cephadm 2024-01-13T08:19:27.987431+0000 mgr.smithi099.oodult (mgr.14180) 79 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi131', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:19:30.150 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:19:30.151 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:19:30.151 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: return f(*arg) 2024-01-13T08:19:30.151 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:19:30.151 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:19:30.151 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:19:30.152 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: code, '\n'.join(err))) 2024-01-13T08:19:30.152 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:19:30.152 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpqu5auolp:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp6ugz_hqr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:19:30.152 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:19:30.152 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:19:30.152 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:19:30.152 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new c5600c88-a4b1-4f42-a8c4-6b1306b1cdeb 2024-01-13T08:19:30.152 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:19:30.152 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:19:30.153 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:19:30.153 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:19:30.153 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:19:30.153 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr self.prepare() 2024-01-13T08:19:30.153 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:30.153 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:30.153 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:19:30.153 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:19:30.154 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:19:30.154 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:19:30.154 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:19:30.154 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:19:30.154 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:19:30.154 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:19:30.154 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:19:30.155 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:19:30.155 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:19:30.155 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:19:30.155 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:19:30.155 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:19:30.155 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:19:30.155 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:19:30.156 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:19:30.156 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:19:30.156 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:19:30.156 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:19:30.156 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:19:30.156 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:19:30.157 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:19:30.157 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:19:30.157 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:19:30.157 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:30.157 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:30.157 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:19:30.157 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:19:30.158 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:19:30.158 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:19:30.158 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:30.158 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:30.158 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:19:30.158 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:19:30.158 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:19:30.158 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:19:30.158 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:19:30.159 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:19:30.159 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:19:30.159 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:19:30.159 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:30.159 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:30.159 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:19:30.159 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:19:30.159 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:19:30.160 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:19:30.160 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:19:30.160 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:19:30.160 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:19:30.160 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: main() 2024-01-13T08:19:30.161 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:19:30.161 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: r = ctx.func(ctx) 2024-01-13T08:19:30.161 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:19:30.161 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:19:30.161 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:19:30.161 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:19:30.162 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:19:30.162 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:19:30.162 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774 2024-01-13T08:19:30.162 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:19:30.162 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:19:30.162 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:19:30.162 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:19:30.162 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:19:30.162 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:19:30.162 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpqu5auolp:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp6ugz_hqr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:19:30.162 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:29 smithi131 conmon[38024]: cluster 2024-01-13T08:19:28.417060+0000 mgr.smithi099.oodult (mgr.14180) 80 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:30.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: cephadm 2024-01-13T08:19:28.828660+0000 mgr.smithi099.oodult (mgr.14180) 81 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi131', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:19:30.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:19:30.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:19:30.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: return f(*arg) 2024-01-13T08:19:30.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:19:30.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:19:30.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:19:30.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: code, '\n'.join(err))) 2024-01-13T08:19:30.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi099/config 2024-01-13T08:19:30.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpb3cveq89:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpxaoq33m0:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:19:30.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:19:30.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:19:30.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:19:30.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 2ef239a7-91e1-4ac5-a0a3-aa7e23b0f567 2024-01-13T08:19:30.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:19:30.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:19:30.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:19:30.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:19:30.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:19:30.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr self.prepare() 2024-01-13T08:19:30.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:30.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:30.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:19:30.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:19:30.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:19:30.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:19:30.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:19:30.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:19:30.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:19:30.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:19:30.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:19:30.984 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:19:30.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:19:30.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:19:30.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:19:30.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:19:30.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:19:30.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:19:30.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:19:30.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:19:30.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:19:30.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:19:30.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:19:30.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:19:30.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:19:30.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:19:30.985 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:19:30.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:30.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:30.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:19:30.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:19:30.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:19:30.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:19:30.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:30.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:30.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:19:30.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:19:30.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:19:30.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:19:30.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:19:30.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:19:30.986 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:19:30.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:19:30.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:30.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:30.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:19:30.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:19:30.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:19:30.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:19:30.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:19:30.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:19:30.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:19:30.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: main() 2024-01-13T08:19:30.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:19:30.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: r = ctx.func(ctx) 2024-01-13T08:19:30.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:19:30.987 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:19:30.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:19:30.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:19:30.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:19:30.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:19:30.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774 2024-01-13T08:19:30.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:19:30.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:19:30.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:19:30.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:19:30.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:19:30.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:19:30.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpb3cveq89:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpxaoq33m0:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:19:30.988 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: cephadm 2024-01-13T08:19:28.829887+0000 mgr.smithi099.oodult (mgr.14180) 82 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T08:19:30.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: service_id: all-available-devices 2024-01-13T08:19:30.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: service_name: osd.all-available-devices 2024-01-13T08:19:30.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: placement: 2024-01-13T08:19:30.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: host_pattern: '*' 2024-01-13T08:19:30.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: spec: 2024-01-13T08:19:30.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: data_devices: 2024-01-13T08:19:30.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: all: true 2024-01-13T08:19:30.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: filter_logic: AND 2024-01-13T08:19:30.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: objectstore: bluestore 2024-01-13T08:19:30.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:19:30.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpqu5auolp:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp6ugz_hqr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:19:30.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:19:30.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:19:30.989 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:19:30.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new c5600c88-a4b1-4f42-a8c4-6b1306b1cdeb 2024-01-13T08:19:30.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:19:30.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:19:30.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:19:30.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:19:30.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:19:30.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr self.prepare() 2024-01-13T08:19:30.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:30.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:30.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:19:30.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:19:30.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:19:30.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:19:30.990 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:19:30.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:19:30.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:19:30.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:19:30.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:19:30.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:19:30.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:19:30.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:19:30.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:19:30.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:19:30.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:19:30.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:19:30.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:19:30.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:19:30.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:19:30.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:19:30.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:19:30.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:19:30.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:19:30.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:19:30.992 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:19:30.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:30.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:30.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:19:30.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:19:30.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:19:30.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:19:30.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:30.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:30.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:19:30.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:19:30.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:19:30.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:19:30.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:19:30.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:19:30.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:19:30.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:19:30.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:30.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:30.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lv 2024-01-13T08:19:30.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: m/zap.py", line 302, in zap_osd 2024-01-13T08:19:30.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:19:30.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:19:30.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:19:30.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:19:30.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:19:30.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:19:30.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: main() 2024-01-13T08:19:30.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:19:30.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: r = ctx.func(ctx) 2024-01-13T08:19:30.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:19:30.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:19:30.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:19:30.995 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:19:30.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:19:30.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:19:30.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:19:30.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:19:30.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:19:30.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:19:30.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:19:30.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:19:30.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpqu5auolp:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp6ugz_hqr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:19:30.996 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:19:30.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T08:19:30.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: if self._apply_service(spec): 2024-01-13T08:19:30.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T08:19:30.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T08:19:30.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T08:19:30.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T08:19:30.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T08:19:30.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T08:19:30.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T08:19:30.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T08:19:30.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T08:19:30.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: raise self._value 2024-01-13T08:19:30.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T08:19:30.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: result = (True, func(*args, **kwds)) 2024-01-13T08:19:30.997 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T08:19:30.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: return list(map(*args)) 2024-01-13T08:19:30.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:19:30.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: return f(*arg) 2024-01-13T08:19:30.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:19:30.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:19:30.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:19:30.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: code, '\n'.join(err))) 2024-01-13T08:19:30.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:19:30.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpqu5auolp:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp6ugz_hqr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:19:30.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:19:30.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:19:30.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:19:30.998 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new c5600c88-a4b1-4f42-a8c4-6b1306b1cdeb 2024-01-13T08:19:30.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:19:30.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:19:30.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:19:30.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:19:30.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:19:30.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr self.prepare() 2024-01-13T08:19:30.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:30.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:30.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:19:30.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:19:30.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:19:30.999 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:19:31.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:19:31.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:19:31.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:19:31.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:19:31.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:19:31.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:19:31.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:19:31.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:19:31.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:19:31.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:19:31.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:19:31.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr F 2024-01-13T08:19:31.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: ile "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:19:31.000 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:19:31.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:19:31.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:19:31.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:19:31.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:19:31.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:19:31.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:19:31.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:19:31.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:19:31.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:31.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:31.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:19:31.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:19:31.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:19:31.001 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:19:31.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:31.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:31.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:19:31.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:19:31.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:19:31.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:19:31.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:19:31.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:19:31.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:19:31.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:19:31.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:31.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:31.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:19:31.002 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:19:31.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:19:31.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:19:31.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:19:31.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:19:31.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:19:31.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: main() 2024-01-13T08:19:31.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:19:31.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: r = ctx.func(ctx) 2024-01-13T08:19:31.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:19:31.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:19:31.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:19:31.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:19:31.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:19:31.003 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:19:31.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:19:31.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:19:31.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:19:31.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:19:31.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:19:31.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:19:31.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpqu5auolp:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp6ugz_hqr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:19:31.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: cluster 2024-01-13T08:19:28.830512+0000 mgr.smithi099.oodult (mgr.14180) 83 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:31.004 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:30 smithi099 conmon[33313]: cluster 2024-01-13T08:19:29.703287+0000 mon.smithi099 (mon.0) 309 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T08:19:31.150 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: cephadm 2024-01-13T08:19:28.828660+0000 mgr.smithi099.oodult (mgr.14180) 81 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi131', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:19:31.150 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:19:31.150 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:19:31.150 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: return f(*arg) 2024-01-13T08:19:31.150 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:19:31.150 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:19:31.150 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:19:31.151 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: code, '\n'.join(err))) 2024-01-13T08:19:31.151 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi099/config 2024-01-13T08:19:31.151 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpb3cveq89:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpxaoq33m0:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:19:31.151 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:19:31.151 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:19:31.151 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:19:31.151 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 2ef239a7-91e1-4ac5-a0a3-aa7e23b0f567 2024-01-13T08:19:31.151 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:19:31.151 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:19:31.151 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:19:31.151 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:19:31.151 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:19:31.152 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr self.prepare() 2024-01-13T08:19:31.152 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:31.152 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:31.152 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:19:31.152 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:19:31.152 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:19:31.152 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:19:31.152 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:19:31.152 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:19:31.153 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:19:31.153 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:19:31.153 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:19:31.153 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:19:31.153 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:19:31.153 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:19:31.153 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:19:31.154 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:19:31.154 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:19:31.154 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:19:31.154 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:19:31.154 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:19:31.154 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:19:31.154 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:19:31.154 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:19:31.154 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:19:31.154 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:19:31.154 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:19:31.155 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:19:31.155 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:31.155 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:31.155 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:19:31.155 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:19:31.155 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:19:31.156 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:19:31.156 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:31.156 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:31.156 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:19:31.156 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:19:31.156 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:19:31.156 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:19:31.156 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:19:31.156 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:19:31.156 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:19:31.156 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:19:31.156 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:31.156 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:31.157 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:19:31.157 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:19:31.157 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:19:31.157 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:19:31.157 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:19:31.157 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:19:31.157 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:19:31.157 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: main() 2024-01-13T08:19:31.157 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:19:31.157 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: r = ctx.func(ctx) 2024-01-13T08:19:31.157 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:19:31.157 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:19:31.157 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:19:31.157 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:19:31.157 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:19:31.158 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:19:31.158 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774 2024-01-13T08:19:31.158 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:19:31.158 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:19:31.158 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:19:31.158 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:19:31.158 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:19:31.158 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:19:31.158 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpb3cveq89:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpxaoq33m0:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:19:31.158 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: cephadm 2024-01-13T08:19:28.829887+0000 mgr.smithi099.oodult (mgr.14180) 82 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T08:19:31.158 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: service_id: all-available-devices 2024-01-13T08:19:31.158 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: service_name: osd.all-available-devices 2024-01-13T08:19:31.158 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: placement: 2024-01-13T08:19:31.158 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: host_pattern: '*' 2024-01-13T08:19:31.159 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: spec: 2024-01-13T08:19:31.159 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: data_devices: 2024-01-13T08:19:31.159 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: all: true 2024-01-13T08:19:31.159 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: filter_logic: AND 2024-01-13T08:19:31.159 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: objectstore: bluestore 2024-01-13T08:19:31.159 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:19:31.159 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpqu5auolp:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp6ugz_hqr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:19:31.159 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:19:31.159 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:19:31.159 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:19:31.160 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new c5600c88-a4b1-4f42-a8c4-6b1306b1cdeb 2024-01-13T08:19:31.161 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:19:31.161 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:19:31.161 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:19:31.161 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:19:31.161 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:19:31.161 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr self.prepare() 2024-01-13T08:19:31.161 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:31.161 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:31.161 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:19:31.161 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:19:31.161 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:19:31.161 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:19:31.161 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:19:31.162 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:19:31.162 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:19:31.162 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:19:31.162 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:19:31.162 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:19:31.162 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:19:31.162 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:19:31.162 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:19:31.162 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:19:31.162 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:19:31.162 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:19:31.162 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:19:31.162 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:19:31.162 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:19:31.163 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:19:31.163 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:19:31.163 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:19:31.163 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:19:31.163 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:19:31.163 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:19:31.163 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:31.163 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:31.163 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:19:31.163 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:19:31.163 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:19:31.163 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:19:31.163 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:31.163 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:31.164 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:19:31.164 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:19:31.164 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:19:31.164 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:19:31.164 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:19:31.164 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:19:31.164 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:19:31.164 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:19:31.164 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:31.164 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:31.164 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lv 2024-01-13T08:19:31.164 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: m/zap.py", line 302, in zap_osd 2024-01-13T08:19:31.164 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:19:31.165 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:19:31.165 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:19:31.165 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:19:31.165 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:19:31.165 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:19:31.165 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: main() 2024-01-13T08:19:31.165 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:19:31.165 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: r = ctx.func(ctx) 2024-01-13T08:19:31.165 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:19:31.165 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:19:31.165 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:19:31.165 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:19:31.165 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:19:31.165 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:19:31.166 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:19:31.166 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:19:31.166 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:19:31.166 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:19:31.166 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:19:31.166 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:19:31.166 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpqu5auolp:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp6ugz_hqr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:19:31.166 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:19:31.166 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T08:19:31.166 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: if self._apply_service(spec): 2024-01-13T08:19:31.166 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T08:19:31.166 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T08:19:31.166 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T08:19:31.166 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T08:19:31.167 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T08:19:31.167 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T08:19:31.167 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T08:19:31.167 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T08:19:31.167 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T08:19:31.167 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: raise self._value 2024-01-13T08:19:31.167 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T08:19:31.167 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: result = (True, func(*args, **kwds)) 2024-01-13T08:19:31.167 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T08:19:31.167 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: return list(map(*args)) 2024-01-13T08:19:31.167 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:19:31.167 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: return f(*arg) 2024-01-13T08:19:31.167 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:19:31.167 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:19:31.167 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:19:31.167 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: code, '\n'.join(err))) 2024-01-13T08:19:31.168 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:19:31.168 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpqu5auolp:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp6ugz_hqr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:19:31.168 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:19:31.168 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:19:31.168 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:19:31.168 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new c5600c88-a4b1-4f42-a8c4-6b1306b1cdeb 2024-01-13T08:19:31.168 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:19:31.168 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:19:31.168 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:19:31.168 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:19:31.168 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:19:31.168 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr self.prepare() 2024-01-13T08:19:31.168 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:31.169 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:31.169 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:19:31.169 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:19:31.169 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:19:31.169 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:19:31.169 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:19:31.169 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:19:31.169 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:19:31.169 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:19:31.169 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:19:31.169 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:19:31.169 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:19:31.169 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:19:31.169 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:19:31.169 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:19:31.170 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:19:31.170 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr F 2024-01-13T08:19:31.170 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: ile "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:19:31.170 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:19:31.170 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:19:31.170 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:19:31.170 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:19:31.170 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:19:31.170 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:19:31.170 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:19:31.170 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:19:31.170 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:19:31.170 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:31.170 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:31.171 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:19:31.171 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:19:31.171 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:19:31.171 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:19:31.171 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:31.172 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:31.172 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:19:31.172 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:19:31.172 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:19:31.172 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:19:31.172 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:19:31.172 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:19:31.173 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:19:31.173 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:19:31.173 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:19:31.173 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:19:31.173 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:19:31.173 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:19:31.173 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:19:31.173 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:19:31.173 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:19:31.173 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:19:31.173 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:19:31.173 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: main() 2024-01-13T08:19:31.173 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:19:31.173 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: r = ctx.func(ctx) 2024-01-13T08:19:31.174 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:19:31.174 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:19:31.174 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:19:31.174 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:19:31.174 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:19:31.174 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:19:31.174 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:19:31.174 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:19:31.174 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:19:31.174 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:19:31.174 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:19:31.174 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:19:31.174 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpqu5auolp:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp6ugz_hqr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:19:31.174 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: cluster 2024-01-13T08:19:28.830512+0000 mgr.smithi099.oodult (mgr.14180) 83 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:31.175 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:30 smithi131 conmon[38024]: cluster 2024-01-13T08:19:29.703287+0000 mon.smithi099 (mon.0) 309 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T08:19:31.765 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:19:32.363 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705133958,"num_remapped_pgs":0} 2024-01-13T08:19:32.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:32 smithi099 conmon[33313]: cluster 2024-01-13T08:19:30.830937+0000 mgr.smithi099.oodult (mgr.14180) 84 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:32.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:32 smithi099 conmon[33313]: audit 2024-01-13T08:19:31.767776+0000 mon.smithi099 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.99:0/3940955285' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:19:33.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:32 smithi131 conmon[38024]: cluster 2024-01-13T08:19:30.830937+0000 mgr.smithi099.oodult (mgr.14180) 84 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:33.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:32 smithi131 conmon[38024]: audit 2024-01-13T08:19:31.767776+0000 mon.smithi099 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.99:0/3940955285' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:19:33.364 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:19:34.887 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:34 smithi099 conmon[33313]: cluster 2024-01-13T08:19:32.831434+0000 mgr.smithi099.oodult (mgr.14180) 2024-01-13T08:19:34.887 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:34 smithi099 conmon[33313]: 85 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:35.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:34 smithi131 conmon[38024]: cluster 2024-01-13T08:19:32.831434+0000 mgr.smithi099.oodult (mgr.14180) 85 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:35.201 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:19:35.813 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705133958,"num_remapped_pgs":0} 2024-01-13T08:19:35.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:35 smithi099 conmon[33313]: audit 2024-01-13T08:19:35.204783+0000 mon.smithi099 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.99:0/1612237673' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:19:36.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:35 smithi131 conmon[38024]: audit 2024-01-13T08:19:35.204783+0000 mon.smithi099 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.99:0/1612237673' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:19:36.815 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:19:36.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:36 smithi099 conmon[33313]: cluster 2024-01-13T08:19:34.831951+0000 mgr.smithi099.oodult (mgr.14180) 86 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:37.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:36 smithi131 conmon[38024]: cluster 2024-01-13T08:19:34.831951+0000 mgr.smithi099.oodult (mgr.14180) 86 2024-01-13T08:19:37.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:36 smithi131 conmon[38024]: : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:38.602 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:19:38.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:38 smithi099 conmon[33313]: cluster 2024-01-13T08:19:36.832552+0000 mgr.smithi099.oodult (mgr.14180) 87 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:38.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:38 smithi099 conmon[33313]: audit 2024-01-13T08:19:38.605624+0000 mon.smithi099 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.99:0/4266138295' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:19:39.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:38 smithi131 conmon[38024]: cluster 2024-01-13T08:19:36.832552+0000 mgr.smithi099.oodult (mgr.14180) 87 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:39.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:38 smithi131 conmon[38024]: audit 2024-01-13T08:19:38.605624+0000 mon.smithi099 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.99:0/4266138295' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:19:39.204 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705133958,"num_remapped_pgs":0} 2024-01-13T08:19:40.206 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:19:40.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:40 smithi099 conmon[33313]: cluster 2024-01-13T08:19:38.833106+0000 mgr.smithi099.oodult (mgr.14180) 88 : cluster 2024-01-13T08:19:40.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:40 smithi099 conmon[33313]: [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:41.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:40 smithi131 conmon[38024]: cluster 2024-01-13T08:19:38.833106+0000 mgr.smithi099.oodult (mgr.14180) 88 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:41.990 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:19:42.519 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705133958,"num_remapped_pgs":0} 2024-01-13T08:19:42.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:42 smithi099 conmon[33313]: cluster 2024-01-13T08:19:40.833665+0000 mgr.smithi099.oodult (mgr.14180) 89 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:42.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:42 smithi099 conmon[33313]: audit 2024-01-13T08:19:41.993457+0000 mon.smithi099 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.99:0/1506236010' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:19:43.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:42 smithi131 conmon[38024]: cluster 2024-01-13T08:19:40.833665+0000 mgr.smithi099.oodult (mgr.14180) 89 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:43.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:42 smithi131 conmon[38024]: audit 2024-01-13T08:19:41.993457+0000 mon.smithi099 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.99:0/1506236010' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:19:43.521 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:19:44.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:44 smithi099 conmon[33313]: cluster 2024-01-13T08:19:42.834098+0000 mgr.smithi099.oodult (mgr.14180) 90 : cluster [DBG] 2024-01-13T08:19:44.991 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:44 smithi099 conmon[33313]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:45.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:44 smithi131 conmon[38024]: cluster 2024-01-13T08:19:42.834098+0000 mgr.smithi099.oodult (mgr.14180) 90 : cluster [DBG] 2024-01-13T08:19:45.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:44 smithi131 conmon[38024]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:45.335 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:19:45.910 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705133958,"num_remapped_pgs":0} 2024-01-13T08:19:46.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:45 smithi131 conmon[38024]: audit 2024-01-13T08:19:45.338521+0000 mon.smithi099 (mon.0) 314 : audit 2024-01-13T08:19:46.150 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:45 smithi131 conmon[38024]: [DBG] from='client.? 172.21.15.99:0/1789675779' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:19:46.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:45 smithi099 conmon[33313]: audit 2024-01-13T08:19:45.338521+0000 mon.smithi099 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.99:0/1789675779' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:19:46.911 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:19:47.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:46 smithi131 conmon[38024]: cluster 2024-01-13T08:19:44.834666+0000 mgr.smithi099.oodult (mgr.14180) 91 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:47.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:46 smithi099 conmon[33313]: cluster 2024-01-13T08:19:44.834666+0000 mgr.smithi099.oodult (mgr.14180) 91 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:47.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:47 smithi099 conmon[33313]: cluster 2024-01-13T08:19:46.835106+0000 mgr.smithi099.oodult (mgr.14180) 92 : cluster 2024-01-13T08:19:47.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:47 smithi099 conmon[33313]: [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:48.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:47 smithi131 conmon[38024]: cluster 2024-01-13T08:19:46.835106+0000 mgr.smithi099.oodult (mgr.14180) 92 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:48.711 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:19:48.993 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:48 smithi099 conmon[33313]: audit 2024-01-13T08:19:48.713776+0000 mon.smithi099 (mon.0 2024-01-13T08:19:48.994 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:48 smithi099 conmon[33313]: ) 315 : audit [DBG] from='client.? 172.21.15.99:0/324487512' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:19:49.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:48 smithi131 conmon[38024]: audit 2024-01-13T08:19:48.713776+0000 mon.smithi099 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.99:0/324487512' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:19:49.284 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705133958,"num_remapped_pgs":0} 2024-01-13T08:19:50.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:49 smithi131 conmon[38024]: cluster 2024-01-13T08:19:48.835509+0000 mgr.smithi099.oodult (mgr.14180) 93 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:50.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:49 smithi099 conmon[33313]: cluster 2024-01-13T08:19:48.835509+0000 mgr.smithi099.oodult (mgr.14180) 2024-01-13T08:19:50.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:49 smithi099 conmon[33313]: 93 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:50.285 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:19:51.969 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:19:52.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:51 smithi099 conmon[33313]: cluster 2024-01-13T08:19:50.835903+0000 mgr.smithi099.oodult (mgr.14180) 94 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:52.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:51 smithi131 conmon[38024]: cluster 2024-01-13T08:19:50.835903+0000 mgr.smithi099.oodult (mgr.14180) 94 : cluster [DBG] 2024-01-13T08:19:52.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:51 smithi131 conmon[38024]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:52.538 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705133958,"num_remapped_pgs":0} 2024-01-13T08:19:53.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:52 smithi099 conmon[33313]: audit 2024-01-13T08:19:51.972632+0000 mon.smithi099 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.99:0/4271409753' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:19:53.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:52 smithi131 conmon[38024]: audit 2024-01-13T08:19:51.972632+0000 mon.smithi099 (mon.0) 316 : audit 2024-01-13T08:19:53.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:52 smithi131 conmon[38024]: [DBG] from='client.? 172.21.15.99:0/4271409753' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:19:53.538 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:19:54.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:53 smithi099 conmon[33313]: cluster 2024-01-13T08:19:52.836357+0000 mgr.smithi099.oodult (mgr.14180) 95 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:54.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:53 smithi131 conmon[38024]: cluster 2024-01-13T08:19:52.836357+0000 mgr.smithi099.oodult (mgr.14180) 95 : cluster 2024-01-13T08:19:54.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:53 smithi131 conmon[38024]: [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:55.303 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:19:55.891 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705133958,"num_remapped_pgs":0} 2024-01-13T08:19:56.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:55 smithi099 conmon[33313]: cluster 2024-01-13T08:19:54.836729+0000 mgr.smithi099.oodult (mgr.14180) 96 : cluster 2024-01-13T08:19:56.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:55 smithi099 conmon[33313]: [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:56.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:55 smithi099 conmon[33313]: audit 2024-01-13T08:19:55.306617+0000 mon.smithi099 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.99:0/1058227604' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:19:56.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:55 smithi131 conmon[38024]: cluster 2024-01-13T08:19:54.836729+0000 mgr.smithi099.oodult (mgr.14180) 96 : cluster [DBG] 2024-01-13T08:19:56.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:55 smithi131 conmon[38024]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:56.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:55 smithi131 conmon[38024]: audit 2024-01-13T08:19:55.306617+0000 mon.smithi099 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.99:0/1058227604' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:19:56.893 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:19:58.334 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:57 smithi099 conmon[33313]: cluster 2024-01-13T08:19:56.837203+0000 mgr.smithi099.oodult (mgr.14180) 97 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:58.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:57 smithi131 conmon[38024]: cluster 2024-01-13T08:19:56.837203+0000 mgr.smithi099.oodult (mgr.14180) 97 : cluster [DBG] 2024-01-13T08:19:58.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:57 smithi131 conmon[38024]: pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:19:58.692 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:19:59.275 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:58 smithi099 conmon[33313]: audit 2024-01-13T08:19:58.695614+0000 mon.smithi099 (mon.0) 318 : audit 2024-01-13T08:19:59.275 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:58 smithi099 conmon[33313]: [DBG] from='client.? 172.21.15.99:0/3312265887' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:19:59.291 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705133958,"num_remapped_pgs":0} 2024-01-13T08:19:59.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:58 smithi131 conmon[38024]: audit 2024-01-13T08:19:58.695614+0000 mon.smithi099 (mon.0) 318 : audit [DBG] 2024-01-13T08:19:59.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:58 smithi131 conmon[38024]: from='client.? 172.21.15.99:0/3312265887' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:20:00.292 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:20:00.322 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:59 smithi099 conmon[33313]: cluster 2024-01-13T08:19:58.837757+0000 mgr.smithi099.oodult (mgr.14180) 98 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:00.322 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:19:59 smithi099 conmon[33313]: 2024-01-13T08:20:00.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:59 smithi131 conmon[38024]: cluster 2024-01-13T08:19:58.837757+0000 mgr.smithi099.oodult (mgr.14180) 98 : cluster [DBG] 2024-01-13T08:20:00.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:19:59 smithi131 conmon[38024]: pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:01.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00.000217+0000 mon.smithi099 (mon.0) 319 : cluster [WRN] Health detail: HEALTH_WARN Failed to apply 1 service(s): osd.all-available-devices 2024-01-13T08:20:01.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00.000265+0000 mon.smithi099 (mon.0) 320 : cluster [WRN] [WRN] CEPHADM_APPLY_SPEC_FAIL: Failed to apply 1 service(s): osd.all-available-devices 2024-01-13T08:20:01.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00.000288+0000 mon.smithi099 (mon.0) 321 : cluster [WRN] osd.all-available-devices: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:20:01.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00.000314+0000 mon.smithi099 (mon.0) 322 : cluster [WRN] Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpqu5auolp:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp6ugz_hqr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:20:01.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00.000343+0000 mon.smithi099 (mon.0) 323 : cluster [WRN] /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:20:01.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00.000368+0000 mon.smithi099 (mon.0) 324 : cluster [WRN] /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:20:01.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00.000391+0000 mon.smithi099 (mon.0) 325 : cluster [WRN] /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:20:01.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00.000417+0000 mon.smithi099 (mon.0) 326 : cluster [WRN] /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new c5600c88-a4b1-4f42-a8c4-6b1306b1cdeb 2024-01-13T08:20:01.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00.000437+0000 mon.smithi099 (mon.0) 327 : cluster 2024-01-13T08:20:01.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: [WRN] /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:20:01.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00.000470+0000 mon.smithi099 (mon.0) 328 : cluster [WRN] /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:20:01.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00.000483+0000 mon.smithi099 (mon.0) 329 : cluster [WRN] /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:20:01.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00.000494+0000 mon.smithi099 (mon.0) 330 : cluster [WRN] /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:20:01.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00.000506+0000 mon.smithi099 (mon.0) 331 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:20:01.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024- 2024-01-13T08:20:01.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 01-13T08:20:00.000517+0000 mon.smithi099 (mon.0) 332 2024-01-13T08:20:01.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: : cluster [WRN] /bin/podman: stderr self.prepare() 2024-01-13T08:20:01.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13 2024-01-13T08:20:01.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: T08:20:00.000538+0000 mon.smithi099 (mon.0) 333 : 2024-01-13T08:20:01.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:01.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00. 2024-01-13T08:20:01.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 000548+0000 mon.smithi099 (mon.0) 334 : cluster [WRN] /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:01.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 2024-01-13T08:20:01.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024- 2024-01-13T08:20:01.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 01-13T08:20:00.000558+0000 mon.smithi099 (mon.0 2024-01-13T08:20:01.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: ) 335 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:20:01.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20 2024-01-13T08:20:01.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: :00.000568+0000 mon.smithi099 (mon.0) 336 : cluster [WRN] 2024-01-13T08:20:01.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:20:01.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00.000590 2024-01-13T08:20:01.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: +0000 mon.smithi099 (mon.0) 337 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:20:01.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 2024-01-13T08:20:01.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:01.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 2024-01-13T08:20:00.000601+0000 mon.smithi099 (mon.0) 338 : cluster 2024-01-13T08:20:01.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: [WRN] /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:20:01.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00.000612 2024-01-13T08:20:01.403 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: +0000 mon.smithi099 (mon.0) 339 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:20:01.403 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13 2024-01-13T08:20:01.403 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: T08:20:00.000629+0000 mon.smithi099 (mon.0) 340 : 2024-01-13T08:20:01.403 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster [WRN] /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:20:01.403 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20: 2024-01-13T08:20:01.403 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 00.000640+0000 mon.smithi099 (mon.0) 341 : cluster [WRN] 2024-01-13T08:20:01.403 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:20:01.403 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024 2024-01-13T08:20:01.403 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: -01-13T08:20:00.000650+0000 mon.smithi099 (mon. 2024-01-13T08:20:01.403 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 0) 342 : cluster [WRN] /bin/podman: stderr 2024-01-13T08:20:01.403 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13 2024-01-13T08:20:01.403 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: T08:20:00.000661+0000 mon.smithi099 (mon.0) 343 : 2024-01-13T08:20:01.403 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster [WRN] /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:20:01.403 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00.000671 2024-01-13T08:20:01.404 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: +0000 mon.smithi099 (mon.0) 344 : cluster [WRN] /bin/podman: stderr 2024-01-13T08:20:01.404 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 2024-01-13T08:20:01.404 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:01.404 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 2024-01-13T08:20:00.000681+0000 mon.smithi099 (mon.0) 345 2024-01-13T08:20:01.404 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: : cluster [WRN] /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:20:01.404 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00. 2024-01-13T08:20:01.404 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 000692+0000 mon.smithi099 (mon.0) 346 : cluster [WRN] /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:20:01.404 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 2024-01-13T08:20:01.404 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:01.404 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 2024-01-13T08:20:00.000702+0000 mon.smithi099 (mon.0 2024-01-13T08:20:01.404 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: ) 347 : cluster [WRN] /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:20:01.404 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20: 2024-01-13T08:20:01.404 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 00.000711+0000 mon.smithi099 (mon.0) 348 : cluster 2024-01-13T08:20:01.404 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:20:01.405 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00.000723+0000 2024-01-13T08:20:01.405 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: mon.smithi099 (mon.0) 349 : cluster [WRN] /bin/podman: stderr self.main(self.argv) 2024-01-13T08:20:01.405 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024 2024-01-13T08:20:01.405 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: -01-13T08:20:00.000742+0000 mon.smithi099 (mon.0 2024-01-13T08:20:01.405 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: ) 350 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:20:01.405 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20 2024-01-13T08:20:01.405 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: :00.000757+0000 mon.smithi099 (mon.0) 351 : cluster 2024-01-13T08:20:01.405 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: [WRN] /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:20:01.407 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00.000767+0000 2024-01-13T08:20:01.407 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: mon.smithi099 (mon.0) 352 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:20:01.407 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13 2024-01-13T08:20:01.407 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: T08:20:00.000782+0000 mon.smithi099 (mon.0) 2024-01-13T08:20:01.407 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 353 : cluster [WRN] /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:20:01.407 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20: 2024-01-13T08:20:01.408 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 00.000794+0000 mon.smithi099 (mon.0) 354 : cluster 2024-01-13T08:20:01.408 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:20:01.408 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00.000804+0000 2024-01-13T08:20:01.408 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: mon.smithi099 (mon.0) 355 : cluster [WRN] /bin/podman: stderr instance.main() 2024-01-13T08:20:01.408 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024- 2024-01-13T08:20:01.408 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 01-13T08:20:00.000817+0000 mon.smithi099 (mon. 2024-01-13T08:20:01.408 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 0) 356 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:20:01.408 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20 2024-01-13T08:20:01.408 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: :00.000827+0000 mon.smithi099 (mon.0) 357 : cluster 2024-01-13T08:20:01.408 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: [WRN] /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:20:01.408 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00.000837+0000 2024-01-13T08:20:01.408 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: mon.smithi099 (mon.0) 358 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:20:01.408 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024- 2024-01-13T08:20:01.408 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 01-13T08:20:00.000848+0000 mon.smithi099 (mon.0 2024-01-13T08:20:01.409 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: ) 359 : cluster [WRN] /bin/podman: stderr instance.main() 2024-01-13T08:20:01.409 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08 2024-01-13T08:20:01.409 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: :20:00.000859+0000 mon.smithi099 (mon.0) 360 : cluster 2024-01-13T08:20:01.409 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:01.409 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00.000869+0000 2024-01-13T08:20:01.409 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: mon.smithi099 (mon.0) 361 : cluster [WRN] /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:01.409 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024- 2024-01-13T08:20:01.409 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 01-13T08:20:00.000879+0000 mon.smithi099 (mon.0) 2024-01-13T08:20:01.409 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 362 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:20:01.409 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20: 2024-01-13T08:20:01.409 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 00.000889+0000 mon.smithi099 (mon.0) 363 : cluster [WRN] /bin/podman: stderr self._execute(plan) 2024-01-13T08:20:01.409 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 2024-01-13T08:20:01.409 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:01.409 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 2024-01-13T08:20:00.000899+0000 mon.smithi099 (mon.0) 364 : 2024-01-13T08:20:01.409 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:20:01.410 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00 2024-01-13T08:20:01.410 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: .000909+0000 mon.smithi099 (mon.0) 365 : cluster [WRN] /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:20:01.410 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 2024-01-13T08:20:01.410 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00.000920+0000 2024-01-13T08:20:01.410 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: mon.smithi099 (mon.0) 366 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:01.410 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024- 2024-01-13T08:20:01.410 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 01-13T08:20:00.000930+0000 mon.smithi099 (mon.0) 2024-01-13T08:20:01.410 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 367 : cluster [WRN] /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:01.410 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20 2024-01-13T08:20:01.410 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: :00.000940+0000 mon.smithi099 (mon.0) 368 : cluster 2024-01-13T08:20:01.410 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:20:01.410 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00. 2024-01-13T08:20:01.410 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 000955+0000 mon.smithi099 (mon.0) 369 : cluster [WRN] 2024-01-13T08:20:01.410 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:20:01.410 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00.000965+0000 2024-01-13T08:20:01.410 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: mon.smithi099 (mon.0) 370 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:20:01.411 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024- 2024-01-13T08:20:01.411 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 01-13T08:20:00.000979+0000 mon.smithi099 (mon. 2024-01-13T08:20:01.411 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 0) 371 : cluster [WRN] /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:20:01.411 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08: 2024-01-13T08:20:01.411 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 20:00.000989+0000 mon.smithi099 (mon.0) 372 : cluster 2024-01-13T08:20:01.411 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:20:01.411 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00. 2024-01-13T08:20:01.411 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 001002+0000 mon.smithi099 (mon.0) 373 : cluster [WRN] 2024-01-13T08:20:01.411 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:20:01.411 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01 2024-01-13T08:20:01.411 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: -13T08:20:00.001012+0000 mon.smithi099 (mon.0) 374 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:20:01.411 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 2024-01-13T08:20:01.411 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:01.411 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 2024-01-13T08:20:00.001022+0000 mon.smithi099 (mon.0) 375 2024-01-13T08:20:01.411 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: : cluster [WRN] /bin/podman: stderr self.zap_osd() 2024-01-13T08:20:01.412 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20 2024-01-13T08:20:01.412 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: :00.001032+0000 mon.smithi099 (mon.0) 376 : cluster [WRN] 2024-01-13T08:20:01.412 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:01.412 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00.001041+0000 mon.smithi099 ( 2024-01-13T08:20:01.412 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: mon.0) 377 : cluster [WRN] /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:01.413 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01- 2024-01-13T08:20:01.413 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 13T08:20:00.001052+0000 mon.smithi099 (mon.0) 378 : cluster [WRN] 2024-01-13T08:20:01.413 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:20:01.413 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00.001062+0000 2024-01-13T08:20:01.413 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: mon.smithi099 (mon.0) 379 : cluster [WRN] /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:20:01.413 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024- 2024-01-13T08:20:01.413 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 01-13T08:20:00.001073+0000 mon.smithi099 (mon.0 2024-01-13T08:20:01.413 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: ) 380 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:20:01.413 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00. 2024-01-13T08:20:01.413 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 001084+0000 mon.smithi099 (mon.0) 381 : cluster [WRN] /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:20:01.413 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 2024-01-13T08:20:01.413 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:01.413 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 2024-01-13T08:20:00.001094+0000 mon.smithi099 (mon.0) 382 : 2024-01-13T08:20:01.413 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster [WRN] /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:20:01.414 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00 2024-01-13T08:20:01.414 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: .001107+0000 mon.smithi099 (mon.0) 383 : cluster [WRN] 2024-01-13T08:20:01.414 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:20:01.414 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:01.414 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 2024-01-13T08:20:00.001125+0000 mon.smithi099 (mon.0) 384 : cluster 2024-01-13T08:20:01.414 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: [WRN] File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:20:01.414 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00.001137+0000 mon.smithi099 ( 2024-01-13T08:20:01.414 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: mon.0) 385 : cluster [WRN] main() 2024-01-13T08:20:01.414 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13 2024-01-13T08:20:01.414 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: T08:20:00.001149+0000 mon.smithi099 (mon.0) 386 2024-01-13T08:20:01.414 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: : cluster [WRN] File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:20:01.414 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00. 2024-01-13T08:20:01.414 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 001175+0000 mon.smithi099 (mon.0) 387 : cluster [WRN] r = ctx.func(ctx) 2024-01-13T08:20:01.415 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01 2024-01-13T08:20:01.415 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: -13T08:20:00.001186+0000 mon.smithi099 (mon.0) 388 2024-01-13T08:20:01.415 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: : cluster [WRN] File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:20:01.415 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20: 2024-01-13T08:20:01.415 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 00.001197+0000 mon.smithi099 (mon.0) 389 : cluster [WRN] 2024-01-13T08:20:01.415 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:20:01.415 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00.001208 2024-01-13T08:20:01.415 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: +0000 mon.smithi099 (mon.0) 390 : cluster [WRN] File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:20:01.415 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024 2024-01-13T08:20:01.415 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: -01-13T08:20:00.001218+0000 mon.smithi099 (mon. 2024-01-13T08:20:01.415 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 0) 391 : cluster [WRN] return func(ctx) 2024-01-13T08:20:01.415 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T 2024-01-13T08:20:01.415 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 08:20:00.001229+0000 mon.smithi099 (mon.0) 392 : 2024-01-13T08:20:01.415 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster [WRN] File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:20:01.415 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00. 2024-01-13T08:20:01.416 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 001244+0000 mon.smithi099 (mon.0) 393 : cluster [WRN] 2024-01-13T08:20:01.416 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:20:01.416 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00.001254+0000 mon.smithi099 2024-01-13T08:20:01.416 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: (mon.0) 394 : cluster [WRN] File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:20:01.416 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024- 2024-01-13T08:20:01.416 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 01-13T08:20:00.001265+0000 mon.smithi099 (mon.0 2024-01-13T08:20:01.416 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: ) 395 : cluster [WRN] return func(ctx) 2024-01-13T08:20:01.416 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00.001274+0000 2024-01-13T08:20:01.416 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: mon.smithi099 (mon.0) 396 : cluster [WRN] File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:20:01.416 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01- 2024-01-13T08:20:01.416 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 13T08:20:00.001284+0000 mon.smithi099 (mon.0) 397 : cluster [WRN] out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:20:01.416 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024- 2024-01-13T08:20:01.416 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 01-13T08:20:00.001294+0000 mon.smithi099 (mon 2024-01-13T08:20:01.417 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: .0) 398 : cluster [WRN] File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:20:01.417 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08 2024-01-13T08:20:01.417 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: :20:00.001304+0000 mon.smithi099 (mon.0) 399 : 2024-01-13T08:20:01.417 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster [WRN] raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:20:01.417 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: cluster 2024-01-13T08:20: 2024-01-13T08:20:01.417 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: 00.001316+0000 mon.smithi099 (mon.0) 400 : cluster [WRN] 2024-01-13T08:20:01.417 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:01 smithi131 conmon[38024]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpqu5auolp:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp6ugz_hqr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:20:01.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00.000217+0000 mon.smithi099 (mon.0) 319 : cluster [WRN] Health detail: HEALTH_WARN Failed to apply 1 service(s): osd.all-available-devices 2024-01-13T08:20:01.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00.000265+0000 mon.smithi099 (mon.0) 320 : cluster [WRN] [WRN] CEPHADM_APPLY_SPEC_FAIL: Failed to apply 1 service(s): osd.all-available-devices 2024-01-13T08:20:01.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00.000288+0000 mon.smithi099 (mon.0) 321 : cluster [WRN] osd.all-available-devices: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:20:01.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00.000314+0000 mon.smithi099 (mon.0) 322 : cluster [WRN] Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpqu5auolp:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp6ugz_hqr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:20:01.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00.000343+0000 mon.smithi099 (mon.0) 323 : cluster [WRN] /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:20:01.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00.000368+0000 mon.smithi099 (mon.0) 324 : cluster [WRN] /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:20:01.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:01.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 2024-01-13T08:20:00.000391+0000 mon.smithi099 (mon.0) 325 : cluster [WRN] /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:20:01.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00.000417+0000 mon.smithi099 (mon.0) 326 : cluster [WRN] /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new c5600c88-a4b1-4f42-a8c4-6b1306b1cdeb 2024-01-13T08:20:01.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00.000437+0000 mon.smithi099 (mon.0) 327 : cluster [WRN] /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:20:01.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00.000470+0000 mon.smithi099 (mon.0) 328 : cluster [WRN] /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:20:01.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00.000483+0000 mon.smithi099 (mon.0) 329 : cluster [WRN] /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:20:01.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00.000494+0000 mon.smithi099 (mon.0) 330 : cluster [WRN] /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:20:01.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00.000506+0000 mon.smithi099 (mon.0) 331 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:20:01.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00.000517+0000 mon.smithi099 (mon.0) 332 : cluster [WRN] /bin/podman: stderr self.prepare() 2024-01-13T08:20:01.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00.000538+0000 mon.smithi099 (mon.0) 333 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:01.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00.000548+0000 mon.smithi099 (mon.0) 334 : cluster [WRN] /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:01.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00.000558+0000 mon.smithi099 (mon.0) 335 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:20:01.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00.000568+0000 mon.smithi099 (mon.0) 336 : cluster [WRN] /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:20:01.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00.000590+0000 mon.smithi099 (mon.0) 337 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:20:01.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00.000601+0000 mon.smithi099 (mon.0) 338 : cluster [WRN] /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:20:01.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00.000612+0000 mon.smithi099 (mon.0) 339 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:20:01.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00.000629+0000 mon.smithi099 (mon.0) 340 : cluster [WRN] /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:20:01.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00.000640+0000 mon.smithi099 (mon.0) 341 : cluster [WRN] /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:20:01.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00.000650+0000 mon.smithi099 (mon.0) 342 : cluster [WRN] /bin/podman: stderr 2024-01-13T08:20:01.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00.000661+0000 mon.smithi099 (mon.0) 343 : cluster [WRN] /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:20:01.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00.000671+0000 mon.smithi099 (mon.0) 344 : cluster [WRN] /bin/podman: stderr 2024-01-13T08:20:01.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00.000681+0000 mon.smithi099 (mon.0) 345 : cluster [WRN] /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:20:01.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20: 2024-01-13T08:20:01.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 00.000692+0000 mon.smithi099 (mon.0) 346 : cluster [WRN] /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:20:01.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00.000702+0000 mon.smithi099 (mon.0) 347 : cluster [WRN] /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:20:01.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00.000711+0000 mon.smithi099 (mon.0) 348 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:20:01.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00.000723+0000 mon.smithi099 (mon 2024-01-13T08:20:01.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: .0) 349 : cluster [WRN] /bin/podman: stderr self.main(self.argv) 2024-01-13T08:20:01.485 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13 2024-01-13T08:20:01.485 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: T08:20:00.000742+0000 mon.smithi099 (mon.0) 350 : cluster [WRN] 2024-01-13T08:20:01.485 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:20:01.485 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00.000757+0000 2024-01-13T08:20:01.486 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: mon.smithi099 (mon.0) 351 : cluster [WRN] /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:20:01.486 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 2024-01-13T08:20:01.486 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:01.486 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 2024-01-13T08:20:00.000767+0000 mon.smithi099 (mon.0 2024-01-13T08:20:01.486 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: ) 352 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:20:01.486 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08: 2024-01-13T08:20:01.486 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 20:00.000782+0000 mon.smithi099 (mon.0) 353 2024-01-13T08:20:01.486 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: : cluster [WRN] /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:20:01.486 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08: 2024-01-13T08:20:01.486 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 20:00.000794+0000 mon.smithi099 (mon.0) 354 2024-01-13T08:20:01.486 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:20:01.486 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20: 2024-01-13T08:20:01.486 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 00.000804+0000 mon.smithi099 (mon.0) 355 : cluster [WRN] 2024-01-13T08:20:01.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:20:01.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:01.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 2024-01-13T08:20:00.000817+0000 mon.smithi099 (mon. 2024-01-13T08:20:01.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 0) 356 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:20:01.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024- 2024-01-13T08:20:01.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 01-13T08:20:00.000827+0000 mon.smithi099 2024-01-13T08:20:01.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: (mon.0) 357 : cluster [WRN] /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:20:01.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024 2024-01-13T08:20:01.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: -01-13T08:20:00.000837+0000 mon.smithi099 2024-01-13T08:20:01.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: (mon.0) 358 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:20:01.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024 2024-01-13T08:20:01.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: -01-13T08:20:00.000848+0000 mon.smithi099 2024-01-13T08:20:01.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: (mon.0) 359 : cluster [WRN] /bin/podman: stderr instance.main() 2024-01-13T08:20:01.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024- 2024-01-13T08:20:01.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 01-13T08:20:00.000859+0000 mon.smithi099 ( 2024-01-13T08:20:01.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: mon.0) 360 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:01.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01 2024-01-13T08:20:01.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: -13T08:20:00.000869+0000 mon.smithi099 (mon 2024-01-13T08:20:01.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: .0) 361 : cluster [WRN] /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:01.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01 2024-01-13T08:20:01.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: -13T08:20:00.000879+0000 mon.smithi099 ( 2024-01-13T08:20:01.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: mon.0) 362 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:20:01.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01- 2024-01-13T08:20:01.489 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 13T08:20:00.000889+0000 mon.smithi099 (mon 2024-01-13T08:20:01.489 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: .0) 363 : cluster [WRN] /bin/podman: stderr self._execute(plan) 2024-01-13T08:20:01.489 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13 2024-01-13T08:20:01.489 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: T08:20:00.000899+0000 mon.smithi099 (mon. 2024-01-13T08:20:01.489 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 0) 364 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:20:01.489 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T 2024-01-13T08:20:01.489 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 08:20:00.000909+0000 mon.smithi099 (mon.0 2024-01-13T08:20:01.489 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: ) 365 : cluster [WRN] /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:20:01.489 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20: 2024-01-13T08:20:01.489 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 00.000920+0000 mon.smithi099 (mon.0) 366 : cluster 2024-01-13T08:20:01.489 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:01.489 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00. 2024-01-13T08:20:01.489 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 000930+0000 mon.smithi099 (mon.0) 367 : cluster [WRN] 2024-01-13T08:20:01.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:01.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00.000940+0000 2024-01-13T08:20:01.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: mon.smithi099 (mon.0) 368 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:20:01.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 2024-01-13T08:20:01.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:01.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 2024-01-13T08:20:00.000955+0000 mon.smithi099 (mon.0 2024-01-13T08:20:01.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: ) 369 : cluster [WRN] /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:20:01.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T 2024-01-13T08:20:01.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 08:20:00.000965+0000 mon.smithi099 (mon.0 2024-01-13T08:20:01.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: ) 370 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:20:01.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T 2024-01-13T08:20:01.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 08:20:00.000979+0000 mon.smithi099 (mon.0 2024-01-13T08:20:01.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: ) 371 : cluster [WRN] /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:20:01.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T 2024-01-13T08:20:01.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 08:20:00.000989+0000 mon.smithi099 (mon.0 2024-01-13T08:20:01.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: ) 372 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:20:01.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T 2024-01-13T08:20:01.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 08:20:00.001002+0000 mon.smithi099 (mon.0 2024-01-13T08:20:01.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: ) 373 : cluster [WRN] /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:20:01.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T 2024-01-13T08:20:01.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 08:20:00.001012+0000 mon.smithi099 (mon.0 2024-01-13T08:20:01.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: ) 374 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:20:01.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024- 2024-01-13T08:20:01.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 01-13T08:20:00.001022+0000 mon.smithi099 ( 2024-01-13T08:20:01.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: mon.0) 375 : cluster [WRN] /bin/podman: stderr self.zap_osd() 2024-01-13T08:20:01.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024- 2024-01-13T08:20:01.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 01-13T08:20:00.001032+0000 mon.smithi099 2024-01-13T08:20:01.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: (mon.0) 376 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:01.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T 2024-01-13T08:20:01.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 08:20:00.001041+0000 mon.smithi099 (mon. 2024-01-13T08:20:01.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 0) 377 : cluster [WRN] /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:01.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13 2024-01-13T08:20:01.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: T08:20:00.001052+0000 mon.smithi099 (mon. 2024-01-13T08:20:01.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 0) 378 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:20:01.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01- 2024-01-13T08:20:01.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 13T08:20:00.001062+0000 mon.smithi099 (mon. 2024-01-13T08:20:01.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 0) 379 : cluster [WRN] /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:20:01.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13 2024-01-13T08:20:01.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: T08:20:00.001073+0000 mon.smithi099 (mon. 2024-01-13T08:20:01.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 0) 380 : cluster [WRN] /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:20:01.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13 2024-01-13T08:20:01.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: T08:20:00.001084+0000 mon.smithi099 (mon. 2024-01-13T08:20:01.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 0) 381 : cluster [WRN] /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:20:01.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13 2024-01-13T08:20:01.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: T08:20:00.001094+0000 mon.smithi099 (mon. 2024-01-13T08:20:01.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 0) 382 : cluster [WRN] /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:20:01.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13 2024-01-13T08:20:01.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: T08:20:00.001107+0000 mon.smithi099 (mon. 2024-01-13T08:20:01.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 0) 383 : cluster [WRN] Traceback (most recent call last): 2024-01-13T08:20:01.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13 2024-01-13T08:20:01.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: T08:20:00.001125+0000 mon.smithi099 (mon. 2024-01-13T08:20:01.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 0) 384 : cluster [WRN] File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:20:01.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T 2024-01-13T08:20:01.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 08:20:00.001137+0000 mon.smithi099 (mon. 2024-01-13T08:20:01.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 0) 385 : cluster [WRN] main() 2024-01-13T08:20:01.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T 2024-01-13T08:20:01.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 08:20:00.001149+0000 mon.smithi099 (mon.0 2024-01-13T08:20:01.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: ) 386 : cluster [WRN] File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:20:01.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08 2024-01-13T08:20:01.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: :20:00.001175+0000 mon.smithi099 (mon.0) 387 : cluster 2024-01-13T08:20:01.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: [WRN] r = ctx.func(ctx) 2024-01-13T08:20:01.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20: 2024-01-13T08:20:01.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 00.001186+0000 mon.smithi099 (mon.0) 388 : cluster 2024-01-13T08:20:01.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: [WRN] File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:20:01.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00. 2024-01-13T08:20:01.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 001197+0000 mon.smithi099 (mon.0) 389 : cluster [WRN] 2024-01-13T08:20:01.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:20:01.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00.001208 2024-01-13T08:20:01.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: +0000 mon.smithi099 (mon.0) 390 : cluster [WRN] 2024-01-13T08:20:01.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:20:01.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00.001218 2024-01-13T08:20:01.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: +0000 mon.smithi099 (mon.0) 391 : cluster [WRN] 2024-01-13T08:20:01.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:20:01.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00. 2024-01-13T08:20:01.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 001229+0000 mon.smithi099 (mon.0) 392 : cluster [WRN] 2024-01-13T08:20:01.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:20:01.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00.001244 2024-01-13T08:20:01.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: +0000 mon.smithi099 (mon.0) 393 : cluster [WRN] return func(ctx) 2024-01-13T08:20:01.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 2024-01-13T08:20:01.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:01.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 2024-01-13T08:20:00.001254+0000 mon.smithi099 (mon. 2024-01-13T08:20:01.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 0) 394 : cluster [WRN] File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:20:01.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:01.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 2024-01-13T08: 2024-01-13T08:20:01.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 20:00.001265+0000 mon.smithi099 (mon.0) 395 : cluster [WRN] 2024-01-13T08:20:01.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:20:01.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20 2024-01-13T08:20:01.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: :00.001274+0000 mon.smithi099 (mon.0) 396 : 2024-01-13T08:20:01.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster [WRN] File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:20:01.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00 2024-01-13T08:20:01.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: .001284+0000 mon.smithi099 (mon.0) 397 : cluster 2024-01-13T08:20:01.496 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: [WRN] out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:20:01.496 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T08:20:01.496 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 2024-01-13T08:20:00.001294+0000 mon.smithi099 (mon.0 2024-01-13T08:20:01.496 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: ) 398 : cluster [WRN] File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:20:01.496 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T 2024-01-13T08:20:01.496 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 08:20:00.001304+0000 mon.smithi099 (mon.0 2024-01-13T08:20:01.496 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: ) 399 : cluster [WRN] raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:20:01.496 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: cluster 2024-01-13T 2024-01-13T08:20:01.496 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 08:20:00.001316+0000 mon.smithi099 (mon.0) 2024-01-13T08:20:01.496 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:01 smithi099 conmon[33313]: 400 : cluster [WRN] RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpqu5auolp:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp6ugz_hqr:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:20:02.035 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:20:02.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:02 smithi131 conmon[38024]: cluster 2024-01-13T08:20:00.838121+0000 mgr.smithi099.oodult (mgr.14180) 99 : 2024-01-13T08:20:02.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:02 smithi131 conmon[38024]: cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:02.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:02 smithi099 conmon[33313]: cluster 2024-01-13T08:20:00.838121+0000 mgr.smithi099.oodult (mgr.14180 2024-01-13T08:20:02.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:02 smithi099 conmon[33313]: ) 99 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:02.641 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705133958,"num_remapped_pgs":0} 2024-01-13T08:20:03.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:03 smithi131 conmon[38024]: audit 2024-01-13T08:20:02.038691+0000 mon.smithi099 (mon.0) 401 : audit 2024-01-13T08:20:03.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:03 smithi131 conmon[38024]: [DBG] from='client.? 172.21.15.99:0/114639793' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:20:03.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:03 smithi099 conmon[33313]: audit 2024-01-13T08:20:02.038691+0000 mon.smithi099 (mon.0) 401 : audit [DBG] from='client.? 172.21.15.99:0/114639793' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:20:03.643 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:20:04.282 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:04 smithi099 conmon[33313]: cluster 2024-01-13T08:20:02.838598+0000 mgr.smithi099.oodult (mgr. 2024-01-13T08:20:04.359 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:04 smithi099 conmon[33313]: 14180) 100 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:04.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:04 smithi131 conmon[38024]: cluster 2024-01-13T08:20:02.838598+0000 mgr.smithi099.oodult (mgr.14180) 2024-01-13T08:20:04.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:04 smithi131 conmon[38024]: 100 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:05.413 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:20:06.080 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705133958,"num_remapped_pgs":0} 2024-01-13T08:20:06.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:06 smithi131 conmon[38024]: cluster 2024-01-13T08:20:04.839104+0000 mgr.smithi099.oodult (mgr.14180) 101 : cluster [DBG] 2024-01-13T08:20:06.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:06 smithi131 conmon[38024]: pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:06.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:06 smithi131 conmon[38024]: audit 2024-01-13T08:20:05.415851+0000 mon.smithi099 (mon.0) 402 : audit [DBG] from='client.? 172.21.15.99:0/3809005811' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:20:06.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:06 smithi099 conmon[33313]: cluster 2024-01-13T08:20:04.839104+0000 mgr.smithi099.oodult (mgr.14180) 101 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:06.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:06 smithi099 conmon[33313]: audit 2024-01-13T08:20:05.415851+0000 mon.smithi099 (mon.0) 402 : audit [DBG] from='client.? 172.21.15.99:0/3809005811' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:20:07.081 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:20:08.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:08 smithi099 conmon[33313]: cluster 2024-01-13T08:20:06.839764+0000 mgr.smithi099.oodult (mgr.14180) 102 : 2024-01-13T08:20:08.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:08 smithi099 conmon[33313]: cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:08.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:08 smithi131 conmon[38024]: cluster 2024-01-13T08:20:06.839764+0000 mgr.smithi099.oodult (mgr.14180) 102 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:08.877 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:20:09.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:09 smithi099 conmon[33313]: audit 2024-01-13T08:20:08.880412+0000 mon.smithi099 (mon.0) 403 : audit [DBG] from='client.? 172.21.15.99:0/3021810938' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:20:09.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:09 smithi131 conmon[38024]: audit 2024-01-13T08:20:08.880412+0000 mon.smithi099 (mon.0) 403 : audit [DBG] 2024-01-13T08:20:09.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:09 smithi131 conmon[38024]: from='client.? 172.21.15.99:0/3021810938' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:20:09.445 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705133958,"num_remapped_pgs":0} 2024-01-13T08:20:10.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:10 smithi131 conmon[38024]: cluster 2024-01-13T08:20:08.840185+0000 mgr.smithi099.oodult (mgr.14180) 103 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:10.447 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:20:10.479 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:10 smithi099 conmon[33313]: cluster 2024-01-13T08:20:08.840185+0000 mgr.smithi099.oodult (mgr.14180) 103 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:12.200 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:20:12.381 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:12 smithi099 conmon[33313]: cluster 2024-01-13T08:20:10.840561+0000 mgr.smithi099.oodult (mgr.14180) 104 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:12.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:12 smithi131 conmon[38024]: cluster 2024-01-13T08:20:10.840561+0000 mgr.smithi099.oodult (mgr.14180) 104 : cluster 2024-01-13T08:20:12.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:12 smithi131 conmon[38024]: [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:12.778 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705133958,"num_remapped_pgs":0} 2024-01-13T08:20:13.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:13 smithi131 conmon[38024]: audit 2024-01-13T08:20:12.203150+0000 mon.smithi099 (mon.0) 404 : audit [DBG] 2024-01-13T08:20:13.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:13 smithi131 conmon[38024]: from='client.? 172.21.15.99:0/1990762605' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:20:13.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:13 smithi099 conmon[33313]: audit 2024-01-13T08:20:12.203150+0000 mon.smithi099 (mon.0) 404 : 2024-01-13T08:20:13.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:13 smithi099 conmon[33313]: audit [DBG] from='client.? 172.21.15.99:0/1990762605' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:20:13.779 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:20:14.229 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:14 smithi099 conmon[33313]: cluster 2024-01-13T08:20:12.840957+0000 mgr.smithi099.oodult (mgr.14180) 105 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:14.229 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:14 smithi099 conmon[33313]: 2024-01-13T08:20:14.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:14 smithi131 conmon[38024]: cluster 2024-01-13T08:20:12.840957+0000 mgr.smithi099.oodult (mgr.14180) 2024-01-13T08:20:14.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:14 smithi131 conmon[38024]: 105 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:15.526 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:20:16.103 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705133958,"num_remapped_pgs":0} 2024-01-13T08:20:16.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:16 smithi131 conmon[38024]: cluster 2024-01-13T08:20:14.841328+0000 mgr.smithi099.oodult (mgr.14180) 106 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:16.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:16 smithi131 conmon[38024]: 2024-01-13T08:20:16.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:16 smithi131 conmon[38024]: audit 2024-01-13T08:20:15.529704+0000 mon.smithi099 (mon.0) 405 : audit [DBG] from='client.? 172.21.15.99:0/2142661224' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:20:16.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:16 smithi099 conmon[33313]: cluster 2024-01-13T08:20:14.841328+0000 mgr.smithi099.oodult (mgr.14180) 106 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:16.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:16 smithi099 conmon[33313]: audit 2024-01-13T08:20:15.529704+0000 mon.smithi099 (mon.0) 405 : audit [DBG] from='client.? 172.21.15.99:0/2142661224' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:20:17.105 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:20:18.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:18 smithi099 conmon[33313]: cluster 2024-01-13T08:20:16.841864+0000 mgr.smithi099.oodult (mgr.14180) 107 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:18.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:18 smithi131 conmon[38024]: cluster 2024-01-13T08:20:16.841864+0000 mgr.smithi099.oodult (mgr.14180) 107 : cluster [DBG] 2024-01-13T08:20:18.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:18 smithi131 conmon[38024]: pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:18.841 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:20:19.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:19 smithi099 conmon[33313]: audit 2024-01-13T08:20:18.844453+0000 mon.smithi099 (mon. 2024-01-13T08:20:19.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:19 smithi099 conmon[33313]: 0) 406 : audit [DBG] from='client.? 172.21.15.99:0/476181364' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:20:19.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:19 smithi131 conmon[38024]: audit 2024-01-13T08:20:18.844453+0000 mon.smithi099 (mon.0) 406 : audit [DBG] 2024-01-13T08:20:19.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:19 smithi131 conmon[38024]: from='client.? 172.21.15.99:0/476181364' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:20:19.471 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705133958,"num_remapped_pgs":0} 2024-01-13T08:20:20.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:20 smithi131 conmon[38024]: cluster 2024-01-13T08:20:18.842232+0000 mgr.smithi099.oodult (mgr.14180) 108 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:20.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:20 smithi131 conmon[38024]: 2024-01-13T08:20:20.472 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:20:20.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:20 smithi099 conmon[33313]: cluster 2024-01-13T08:20:18.842232+0000 mgr.smithi099.oodult (mgr.14180) 108 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:20.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:20 smithi099 conmon[33313]: 2024-01-13T08:20:22.173 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:20:22.387 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:22 smithi099 conmon[33313]: cluster 2024-01-13T08:20:20.842654+0000 mgr.smithi099.oodult (mgr.14180) 109 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:22.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:22 smithi131 conmon[38024]: cluster 2024-01-13T08:20:20.842654+0000 mgr.smithi099.oodult (mgr.14180) 109 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:22.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:22 smithi131 conmon[38024]: 2024-01-13T08:20:22.760 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705133958,"num_remapped_pgs":0} 2024-01-13T08:20:23.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:23 smithi131 conmon[38024]: audit 2024-01-13T08:20:22.176905+0000 mon.smithi099 (mon.0) 407 : audit 2024-01-13T08:20:23.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:23 smithi131 conmon[38024]: [DBG] from='client.? 172.21.15.99:0/3243778862' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:20:23.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:23 smithi099 conmon[33313]: audit 2024-01-13T08:20:22.176905+0000 mon.smithi099 (mon.0) 407 : audit [DBG] from='client.? 172.21.15.99:0/3243778862' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:20:23.761 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:20:24.374 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:24 smithi099 conmon[33313]: cluster 2024-01-13T08:20:22.843036+0000 mgr.smithi099.oodult (mgr.14180) 110 : cluster [DBG] 2024-01-13T08:20:24.375 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:24 smithi099 conmon[33313]: pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:24.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:24 smithi131 conmon[38024]: cluster 2024-01-13T08:20:22.843036+0000 2024-01-13T08:20:24.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:24 smithi131 conmon[38024]: mgr.smithi099.oodult (mgr.14180) 110 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:25.500 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:20:26.087 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705133958,"num_remapped_pgs":0} 2024-01-13T08:20:26.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:26 smithi131 conmon[38024]: cluster 2024-01-13T08:20:24.843433+0000 mgr.smithi099.oodult (mgr.14180) 111 : cluster [DBG] 2024-01-13T08:20:26.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:26 smithi131 conmon[38024]: pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:26.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:26 smithi131 conmon[38024]: audit 2024-01-13T08:20:25.504385+0000 mon.smithi099 (mon.0) 408 : audit [DBG] from='client.? 172.21.15.99:0/3344333054' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:20:26.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:26 smithi099 conmon[33313]: cluster 2024-01-13T08:20:24.843433+0000 mgr.smithi099.oodult (mgr.14180) 111 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:26.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:26 smithi099 conmon[33313]: audit 2024-01-13T08:20:25.504385+0000 mon.smithi099 (mon.0) 408 : audit [DBG] from='client.? 172.21.15.99:0/3344333054' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:20:27.089 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:20:28.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:28 smithi099 conmon[33313]: cluster 2024-01-13T08:20:26.843918+0000 mgr.smithi099.oodult (mgr.14180) 112 : cluster 2024-01-13T08:20:28.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:28 smithi099 conmon[33313]: [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:28.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:28 smithi131 conmon[38024]: cluster 2024-01-13T08:20:26.843918+0000 mgr.smithi099.oodult (mgr.14180) 112 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:28.838 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:20:29.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:29 smithi099 conmon[33313]: audit 2024-01-13T08:20:28.831352+0000 mon.smithi099 (mon.0) 409 2024-01-13T08:20:29.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:29 smithi099 conmon[33313]: : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:20:29.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:29 smithi099 conmon[33313]: audit 2024-01-13T08:20:28.842445+0000 mon.smithi099 (mon.0) 410 : audit [DBG] from='client.? 172.21.15.99:0/4169910362' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:20:29.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:29 smithi131 conmon[38024]: audit 2024-01-13T08:20:28.831352+0000 mon.smithi099 (mon.0) 409 : 2024-01-13T08:20:29.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:29 smithi131 conmon[38024]: audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:20:29.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:29 smithi131 conmon[38024]: audit 2024-01-13T08:20:28.842445+0000 mon.smithi099 (mon.0) 410 : audit [DBG] from='client.? 172.21.15.99:0/4169910362' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:20:29.433 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705133958,"num_remapped_pgs":0} 2024-01-13T08:20:30.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:30 smithi131 conmon[38024]: cluster 2024-01-13T08:20:28.844322+0000 mgr.smithi099.oodult (mgr.14180) 113 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:30.434 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:20:30.461 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:30 smithi099 conmon[33313]: cluster 2024-01-13T08:20:28.844322+0000 mgr.smithi099.oodult (mgr.14180) 2024-01-13T08:20:30.461 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:30 smithi099 conmon[33313]: 113 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:31.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:31 smithi131 conmon[38024]: audit 2024-01-13T08:20:30.142860+0000 mon.smithi099 (mon.0) 411 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:20:31.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:31 smithi099 conmon[33313]: audit 2024-01-13T08:20:30.142860+0000 mon.smithi099 (mon.0) 2024-01-13T08:20:31.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:31 smithi099 conmon[33313]: 411 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:20:32.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:32 smithi131 conmon[38024]: cluster 2024-01-13T08:20:30.844754+0000 mgr.smithi099.oodult (mgr.14180) 114 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:32.424 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:32 smithi099 conmon[33313]: cluster 2024-01-13T08:20:30.844754+0000 mgr.smithi099.oodult (mgr.14180) 114 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:32.789 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:20:33.400 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":13,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705133958,"num_remapped_pgs":0} 2024-01-13T08:20:34.109 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:33 smithi099 conmon[33313]: audit 2024-01-13T08:20:32.768437+0000 mon.smithi099 (mon.0) 412 : 2024-01-13T08:20:34.110 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:33 smithi099 conmon[33313]: audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:20:34.110 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:33 smithi099 conmon[33313]: audit 2024-01-13T08:20:32.793165+0000 mon.smithi099 (mon.0) 413 : audit [DBG] from='client.? 172.21.15.99:0/4036904786' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:20:34.110 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:33 smithi099 conmon[33313]: cluster 2024-01-13T08:20:32.845273+0000 mgr.smithi099.oodult (mgr.14180) 115 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:34.110 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:33 smithi099 conmon[33313]: audit 2024-01-13T08:20:33.300981+0000 mon.smithi099 (mon.0) 414 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:20:34.110 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:33 smithi099 conmon[33313]: audit 2024-01-13T08:20:33.301668+0000 mon.smithi099 (mon.0) 415 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T08:20:34.110 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:33 smithi099 conmon[33313]: audit 2024-01-13T08:20:33.306145+0000 mon.smithi099 (mon.0) 416 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:20:34.110 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:33 smithi099 conmon[33313]: cluster 2024-01-13T08:20:33.306470+0000 mgr.smithi099.oodult (mgr.14180) 116 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:34.110 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:33 smithi099 conmon[33313]: audit 2024-01-13T08:20:33.310033+0000 mon.smithi099 (mon.0) 417 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T08:20:34.110 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:33 smithi099 conmon[33313]: audit 2024-01-13T08:20:33.312710+0000 mon.smithi099 (mon.0) 418 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:20:34.110 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:33 smithi099 conmon[33313]: audit 2024-01-13T08:20:33.313616+0000 mon.smithi099 (mon.0) 419 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:20:34.110 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:33 smithi099 conmon[33313]: audit 2024-01-13T08:20:33.313815+0000 mon.smithi099 (mon.0) 420 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:20:34.110 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:33 smithi099 conmon[33313]: audit 2024-01-13T08:20:33.314240+0000 mon.smithi099 (mon.0) 421 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:20:34.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:33 smithi131 conmon[38024]: audit 2024-01-13T08:20:32.768437+0000 mon.smithi099 (mon.0) 412 2024-01-13T08:20:34.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:33 smithi131 conmon[38024]: : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:20:34.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:33 smithi131 conmon[38024]: audit 2024-01-13T08:20:32.793165+0000 mon.smithi099 (mon.0) 413 : audit [DBG] from='client.? 172.21.15.99:0/4036904786' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:20:34.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:33 smithi131 conmon[38024]: cluster 2024-01-13T08:20:32.845273+0000 mgr.smithi099.oodult (mgr.14180) 115 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:34.150 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:33 smithi131 conmon[38024]: audit 2024-01-13T08:20:33.300981+0000 mon.smithi099 (mon.0) 414 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:20:34.150 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:33 smithi131 conmon[38024]: audit 2024-01-13T08:20:33.301668+0000 mon.smithi099 (mon.0) 415 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T08:20:34.150 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:33 smithi131 conmon[38024]: audit 2024-01-13T08:20:33.306145+0000 mon.smithi099 (mon.0) 416 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:20:34.150 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:33 smithi131 conmon[38024]: 2024-01-13T08:20:34.150 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:33 smithi131 conmon[38024]: cluster 2024-01-13T08:20:33.306470+0000 mgr.smithi099.oodult (mgr.14180) 116 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:34.150 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:33 smithi131 conmon[38024]: audit 2024-01-13T08:20:33.310033+0000 mon.smithi099 (mon. 2024-01-13T08:20:34.150 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:33 smithi131 conmon[38024]: 0) 417 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T08:20:34.150 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:33 smithi131 conmon[38024]: audit 2024-01-13T08:20:34.150 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:33 smithi131 conmon[38024]: 2024-01-13T08:20:33. 2024-01-13T08:20:34.150 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:33 smithi131 conmon[38024]: 312710+0000 mon.smithi099 (mon.0) 418 : audit 2024-01-13T08:20:34.150 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:33 smithi131 conmon[38024]: [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:20:34.150 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:33 smithi131 conmon[38024]: audit 2024-01-13T08:20:33. 2024-01-13T08:20:34.150 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:33 smithi131 conmon[38024]: 313616+0000 mon.smithi099 (mon.0) 419 : audit 2024-01-13T08:20:34.150 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:33 smithi131 conmon[38024]: [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:20:34.150 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:33 smithi131 conmon[38024]: audit 2024-01-13T08:20:33.313815+0000 2024-01-13T08:20:34.151 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:33 smithi131 conmon[38024]: mon.smithi099 (mon.0) 420 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:20:34.151 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:33 smithi131 conmon[38024]: audit 2024-01-13T08:20:34.151 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:33 smithi131 conmon[38024]: 2024-01-13T08: 2024-01-13T08:20:34.151 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:33 smithi131 conmon[38024]: 20:33.314240+0000 mon.smithi099 (mon.0) 421 2024-01-13T08:20:34.151 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:33 smithi131 conmon[38024]: : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:20:34.402 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:20:34.852 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:34 smithi131 conmon[38024]: cluster 2024-01-13T08:20:33.766636+0000 mon.smithi099 (mon.0) 422 : cluster 2024-01-13T08:20:34.865 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:34 smithi131 conmon[38024]: [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T08:20:34.865 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:34 smithi131 conmon[38024]: cluster 2024-01-13T08:20:33.766687+0000 mon.smithi099 (mon.0) 423 : cluster [INF] Cluster is now healthy 2024-01-13T08:20:34.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:34 smithi099 conmon[33313]: cluster 2024-01-13T08:20:33.766636+0000 mon.smithi099 (mon.0) 422 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T08:20:34.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:34 smithi099 conmon[33313]: cluster 2024-01-13T08:20:33.766687+0000 mon.smithi099 (mon.0) 423 : cluster [INF] Cluster is now healthy 2024-01-13T08:20:36.054 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:35 smithi099 conmon[33313]: cluster 2024-01-13T08:20:35.306658+0000 mgr.smithi099.oodult (mgr.14180) 117 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:36.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:35 smithi131 conmon[38024]: cluster 2024-01-13T08:20:35.306658+0000 mgr.smithi099.oodult (mgr.14180) 117 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:36.424 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:20:37.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: audit 2024-01-13T08:20:35.874694+0000 mon.smithi099 (mon.0) 2024-01-13T08:20:37.150 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: 424 : audit [INF] from='client.? 172.21.15.99:0/2308219448' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91656956-5a42-4abe-ab0b-485172d8a222"}]: dispatch 2024-01-13T08:20:37.150 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: audit 2024-01-13T08:20:35.878496+0000 mon.smithi099 (mon.0) 425 : audit [INF] from='client.? 172.21.15.99:0/2308219448' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91656956-5a42-4abe-ab0b-485172d8a222"}]': finished 2024-01-13T08:20:37.150 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: cluster 2024-01-13T08:20:35.878538+0000 mon.smithi099 (mon.0) 426 : cluster [DBG] osdmap e14: 1 total, 0 up, 1 in 2024-01-13T08:20:37.150 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: audit 2024-01-13T08:20:35.878625+0000 mon.smithi099 (mon.0) 427 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:20:37.150 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: audit 2024-01-13T08: 2024-01-13T08:20:37.150 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: 20:35.918213+0000 mon.smithi131 (mon.1) 6 : audit [INF] from='client.? 172.21.15.131:0/4052372371' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "96221318-0687-4c79-a5f1-fc164398f2b6"}]: dispatch 2024-01-13T08:20:37.150 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: audit 2024-01-13T08:20:35. 2024-01-13T08:20:37.150 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: 918582+0000 mon.smithi099 (mon.0) 428 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "96221318-0687-4c79-a5f1-fc164398f2b6"}]: dispatch 2024-01-13T08:20:37.150 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: audit 2024-01-13T 2024-01-13T08:20:37.150 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: 08:20:35.921928+0000 mon.smithi099 (mon.0 2024-01-13T08:20:37.150 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: ) 429 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "96221318-0687-4c79-a5f1-fc164398f2b6"}]': finished 2024-01-13T08:20:37.151 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: cluster 2024-01-13T 2024-01-13T08:20:37.151 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: 08:20:35.921980+0000 mon.smithi099 (mon.0 2024-01-13T08:20:37.151 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: ) 430 : cluster [DBG] osdmap e15: 2 total, 0 up, 2 in 2024-01-13T08:20:37.151 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: audit 2024-01-13T08 2024-01-13T08:20:37.151 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: :20:35.922074+0000 mon.smithi099 (mon.0) 2024-01-13T08:20:37.151 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: 431 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:20:37.151 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: audit 2024-01-13T08 2024-01-13T08:20:37.151 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: :20:35.922338+0000 mon.smithi099 (mon.0) 432 2024-01-13T08:20:37.151 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-13T08:20:37.151 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: audit 2024-01-13T08: 2024-01-13T08:20:37.151 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: 20:36.185256+0000 mon.smithi099 (mon.0) 433 2024-01-13T08:20:37.151 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: : audit [INF] from='client.? 172.21.15.99:0/212077627' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:20:37.151 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: audit 2024-01-13T08:20 2024-01-13T08:20:37.151 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: :36.188955+0000 mon.smithi099 (mon.0) 434 : audit [INF] from='client.? 172.21.15.99:0/212077627' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:20:37.151 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: 2024-01-13T08:20:37.152 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: cluster 2024-01-13T08:20:37.152 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: 2024-01-13T08:20:36.188994+0000 mon.smithi099 (mon.0 2024-01-13T08:20:37.152 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: ) 435 : cluster [DBG] osdmap e16: 1 total, 0 up, 1 in 2024-01-13T08:20:37.152 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: audit 2024-01-13 2024-01-13T08:20:37.152 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: T08:20:36.189100+0000 mon.smithi099 (mon.0) 2024-01-13T08:20:37.152 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: 436 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-13T08:20:37.152 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: audit 2024-01-13T08:20 2024-01-13T08:20:37.152 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: :36.211491+0000 mon.smithi131 (mon.1) 7 : 2024-01-13T08:20:37.152 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: audit [INF] from='client.? 172.21.15.131:0/3086859697' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:20:37.152 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: audit 2024-01-13T08:20:36 2024-01-13T08:20:37.152 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: .211856+0000 mon.smithi099 (mon.0) 437 : audit 2024-01-13T08:20:37.152 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:20:37.152 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: audit 2024-01-13T08:20:36. 2024-01-13T08:20:37.152 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: 215359+0000 mon.smithi099 (mon.0) 438 : audit [INF] 2024-01-13T08:20:37.153 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:20:37.153 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: cluster 2024-01-13T08:20:36.215430 2024-01-13T08:20:37.153 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: +0000 mon.smithi099 (mon.0) 439 : cluster [DBG] 2024-01-13T08:20:37.153 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: osdmap e17: 0 total, 0 up, 0 in 2024-01-13T08:20:37.153 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: audit 2024-01-13T08:20:36.428367 2024-01-13T08:20:37.153 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:36 smithi131 conmon[38024]: +0000 mon.smithi099 (mon.0) 440 : audit [DBG] from='client.? 172.21.15.99:0/3759387061' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:20:37.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:36 smithi099 conmon[33313]: audit 2024-01-13T08:20:35.874694+0000 mon.smithi099 (mon.0) 424 : audit [INF] from='client.? 172.21.15.99:0/2308219448' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91656956-5a42-4abe-ab0b-485172d8a222"}]: dispatch 2024-01-13T08:20:37.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:36 smithi099 conmon[33313]: 2024-01-13T08:20:37.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:36 smithi099 conmon[33313]: audit 2024-01-13T08:20:35.878496+0000 mon.smithi099 (mon.0) 425 : audit [INF] from='client.? 172.21.15.99:0/2308219448' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91656956-5a42-4abe-ab0b-485172d8a222"}]': finished 2024-01-13T08:20:37.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:36 smithi099 conmon[33313]: cluster 2024-01-13T08:20:35.878538+0000 mon.smithi099 (mon.0) 426 : cluster [DBG] osdmap e14: 1 total, 0 up, 1 in 2024-01-13T08:20:37.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:36 smithi099 conmon[33313]: audit 2024-01-13T08:20:35.878625+0000 mon.smithi099 (mon.0) 427 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:20:37.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:36 smithi099 conmon[33313]: audit 2024-01-13T08:20:35.918213+0000 mon.smithi131 (mon.1) 6 : audit [INF] from='client.? 172.21.15.131:0/4052372371' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "96221318-0687-4c79-a5f1-fc164398f2b6"}]: dispatch 2024-01-13T08:20:37.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:36 smithi099 conmon[33313]: audit 2024-01-13T08:20:35.918582+0000 mon.smithi099 (mon.0) 428 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "96221318-0687-4c79-a5f1-fc164398f2b6"}]: dispatch 2024-01-13T08:20:37.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:36 smithi099 conmon[33313]: audit 2024-01-13T08:20:35.921928+0000 mon.smithi099 (mon.0) 429 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "96221318-0687-4c79-a5f1-fc164398f2b6"}]': finished 2024-01-13T08:20:37.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:36 smithi099 conmon[33313]: cluster 2024-01-13T08:20:35.921980+0000 mon.smithi099 (mon.0) 430 : cluster [DBG] osdmap e15: 2 total, 0 up, 2 in 2024-01-13T08:20:37.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:36 smithi099 conmon[33313]: audit 2024-01-13T08:20:35.922074+0000 mon.smithi099 (mon.0) 431 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:20:37.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:36 smithi099 conmon[33313]: audit 2024-01-13T08:20:35.922338+0000 mon.smithi099 (mon.0) 432 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-13T08:20:37.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:36 smithi099 conmon[33313]: audit 2024-01-13T08:20:36.185256+0000 mon.smithi099 (mon.0) 433 : audit [INF] from='client.? 172.21.15.99:0/212077627' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:20:37.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:36 smithi099 conmon[33313]: audit 2024-01-13T08:20:36.188955+0000 mon.smithi099 (mon.0) 434 : audit [INF] from='client.? 172.21.15.99:0/212077627' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:20:37.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:36 smithi099 conmon[33313]: cluster 2024-01-13T08:20:36.188994+0000 mon.smithi099 (mon.0) 435 : cluster [DBG] osdmap e16: 1 total, 0 up, 1 in 2024-01-13T08:20:37.233 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:36 smithi099 conmon[33313]: audit 2024-01-13T08:20:36.189100+0000 mon.smithi099 (mon.0) 436 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-13T08:20:37.233 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:36 smithi099 conmon[33313]: audit 2024-01-13T08:20:36.211491+0000 mon.smithi131 (mon.1) 7 : audit [INF] from='client.? 172.21.15.131:0/3086859697' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:20:37.233 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:36 smithi099 conmon[33313]: audit 2024-01-13T08:20:36.211856+0000 mon.smithi099 (mon.0) 437 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:20:37.233 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:36 smithi099 conmon[33313]: audit 2024-01-13T08:20:36.215359+0000 mon.smithi099 (mon.0) 438 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:20:37.233 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:36 smithi099 conmon[33313]: cluster 2024-01-13T08:20:36.215430+0000 mon.smithi099 (mon.0) 439 : cluster [DBG] osdmap e17: 0 total, 0 up, 0 in 2024-01-13T08:20:37.233 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:36 smithi099 conmon[33313]: audit 2024-01-13T08:20:36.428367+0000 mon.smithi099 (mon.0) 440 : audit [DBG] from='client.? 172.21.15.99:0/3759387061' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:20:37.291 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:20:38.292 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:20:38.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: cephadm 2024-01-13T08:20:36.936187+0000 mgr.smithi099.oodult (mgr.14180) 118 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi131', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:20:38.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:20:38.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:20:38.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: return f(*arg) 2024-01-13T08:20:38.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:20:38.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:20:38.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:20:38.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: code, '\n'.join(err))) 2024-01-13T08:20:38.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:20:38.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpkfbnjqwi:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp04zvn9e0:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:20:38.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:20:38.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:20:38.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:20:38.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 96221318-0687-4c79-a5f1-fc164398f2b6 2024-01-13T08:20:38.652 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:20:38.652 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 --yes-i-really-mean-it 2024-01-13T08:20:38.652 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr stderr: purged osd.1 2024-01-13T08:20:38.652 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:20:38.652 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:20:38.652 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr self.prepare() 2024-01-13T08:20:38.652 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:38.652 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:38.652 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:20:38.652 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:20:38.653 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:20:38.653 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:20:38.653 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:20:38.653 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:20:38.653 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:20:38.653 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:20:38.653 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:20:38.653 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:20:38.653 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:20:38.653 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:20:38.653 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:20:38.653 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:20:38.653 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:20:38.654 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:20:38.654 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:20:38.654 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:20:38.654 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:20:38.654 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:20:38.654 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:20:38.654 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:20:38.654 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:20:38.655 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:20:38.655 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:20:38.655 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:38.655 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:38.655 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:20:38.655 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:20:38.655 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:20:38.655 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:20:38.655 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:38.655 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:38.655 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:20:38.655 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:20:38.655 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:20:38.655 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:20:38.656 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:20:38.656 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:20:38.656 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:20:38.656 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:20:38.656 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:38.656 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:38.657 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:20:38.657 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:20:38.657 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:20:38.657 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:20:38.657 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 1 2024-01-13T08:20:38.657 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:20:38.658 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:20:38.658 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: main() 2024-01-13T08:20:38.658 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:20:38.658 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: r = ctx.func(ctx) 2024-01-13T08:20:38.658 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:20:38.658 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:20:38.658 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:20:38.658 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:20:38.658 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:20:38.658 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:20:38.658 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-8777 2024-01-13T08:20:38.658 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: 4f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:20:38.658 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:20:38.658 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:20:38.659 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:20:38.659 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:20:38.659 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:20:38.659 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpkfbnjqwi:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp04zvn9e0:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:20:38.659 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: audit 2024-01-13T08:20:37.246251+0000 mon.smithi099 (mon.0) 441 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:20:38.659 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: cephadm 2024-01-13T08:20:37.270306+0000 mgr.smithi099.oodult (mgr.14180) 119 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi131', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:20:38.659 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:20:38.659 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:20:38.659 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: return f(*arg) 2024-01-13T08:20:38.659 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:20:38.659 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:20:38.659 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:20:38.659 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: code, '\n'.join(err))) 2024-01-13T08:20:38.659 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi099/config 2024-01-13T08:20:38.660 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpkfqs4er9:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmptgqnpszv:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:20:38.660 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:20:38.660 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:20:38.660 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:20:38.660 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 91656956-5a42-4abe-ab0b-485172d8a222 2024-01-13T08:20:38.660 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:20:38.660 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:20:38.660 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:20:38.660 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:20:38.660 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:20:38.660 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr self.prepare() 2024-01-13T08:20:38.660 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:38.660 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:38.660 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:20:38.661 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:20:38.661 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:20:38.661 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:20:38.661 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:20:38.661 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:20:38.661 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:20:38.661 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:20:38.661 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:20:38.661 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:20:38.661 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:20:38.661 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:20:38.661 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:20:38.661 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:20:38.661 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:20:38.662 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:20:38.662 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:20:38.662 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:20:38.662 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:20:38.662 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:20:38.663 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:20:38.663 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:20:38.663 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:20:38.664 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:20:38.664 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:20:38.664 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:38.664 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:38.664 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:20:38.664 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:20:38.664 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:20:38.664 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:20:38.664 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:38.664 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:38.664 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:20:38.664 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:20:38.664 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:20:38.665 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr r 2024-01-13T08:20:38.665 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: ollback_osd(self.args, self.osd_id) 2024-01-13T08:20:38.665 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:20:38.665 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:20:38.665 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:20:38.665 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:20:38.665 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:38.665 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:38.665 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:20:38.665 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:20:38.665 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:20:38.665 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:20:38.665 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:20:38.665 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:20:38.666 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:20:38.666 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: main() 2024-01-13T08:20:38.666 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:20:38.666 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: r = ctx.func(ctx) 2024-01-13T08:20:38.666 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:20:38.666 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:20:38.666 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:20:38.666 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:20:38.666 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:20:38.666 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:20:38.666 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:20:38.666 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:20:38.666 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:20:38.666 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:20:38.666 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:20:38.667 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:20:38.667 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpkfqs4er9:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmptgqnpszv:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:20:38.667 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: cephadm 2024-01-13T08:20:37.271742+0000 mgr.smithi099.oodult (mgr.14180) 120 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T08:20:38.667 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: service_id: all-available-devices 2024-01-13T08:20:38.667 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: service_name: osd.all-available-devices 2024-01-13T08:20:38.667 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: placement: 2024-01-13T08:20:38.667 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: host_pattern: '*' 2024-01-13T08:20:38.667 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: spec: 2024-01-13T08:20:38.668 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: data_devices: 2024-01-13T08:20:38.668 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: all: true 2024-01-13T08:20:38.668 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: filter_logic: AND 2024-01-13T08:20:38.668 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: objectstore: bluestore 2024-01-13T08:20:38.668 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:20:38.668 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpkfbnjqwi:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp04zvn9e0:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:20:38.668 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:20:38.668 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:20:38.668 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:20:38.668 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 96221318-0687-4c79-a5f1-fc164398f2b6 2024-01-13T08:20:38.669 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:20:38.669 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 --yes-i-really-mean-it 2024-01-13T08:20:38.669 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr stderr: purged osd.1 2024-01-13T08:20:38.669 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:20:38.669 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:20:38.669 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr self.prepare() 2024-01-13T08:20:38.669 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:38.669 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:38.669 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:20:38.669 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:20:38.669 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:20:38.669 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:20:38.669 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:20:38.669 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:20:38.669 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:20:38.670 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:20:38.670 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:20:38.670 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:20:38.670 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:20:38.670 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:20:38.670 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:20:38.670 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:20:38.670 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:20:38.670 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:20:38.670 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:20:38.670 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:20:38.670 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:20:38.670 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py 2024-01-13T08:20:38.670 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: ", line 194, in dispatch 2024-01-13T08:20:38.671 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:20:38.671 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:20:38.671 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:20:38.671 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:20:38.671 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:20:38.671 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:38.671 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:38.671 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:20:38.671 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:20:38.671 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:20:38.671 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:20:38.671 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:38.671 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:38.671 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:20:38.671 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:20:38.672 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:20:38.672 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:20:38.672 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:20:38.672 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:20:38.672 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:20:38.672 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:20:38.672 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:38.672 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:38.672 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:20:38.672 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:20:38.672 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:20:38.672 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:20:38.673 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 1 2024-01-13T08:20:38.673 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:20:38.673 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:20:38.673 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: main() 2024-01-13T08:20:38.673 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:20:38.673 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: r = ctx.func(ctx) 2024-01-13T08:20:38.673 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:20:38.673 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:20:38.673 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:20:38.673 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:20:38.673 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:20:38.673 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:20:38.673 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:20:38.673 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:20:38.674 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:20:38.674 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:20:38.674 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:20:38.674 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:20:38.674 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpkfbnjqwi:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp04zvn9e0:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:20:38.674 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:20:38.674 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T08:20:38.674 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: if self._apply_service(spec): 2024-01-13T08:20:38.674 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T08:20:38.674 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T08:20:38.674 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T08:20:38.674 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T08:20:38.674 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T08:20:38.674 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T08:20:38.675 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T08:20:38.675 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T08:20:38.675 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T08:20:38.675 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: raise self._value 2024-01-13T08:20:38.675 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T08:20:38.675 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: result = (True, func(*args, **kwds)) 2024-01-13T08:20:38.675 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T08:20:38.675 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: return list(map(*args)) 2024-01-13T08:20:38.675 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:20:38.675 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: return f(*arg) 2024-01-13T08:20:38.675 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:20:38.675 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:20:38.675 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:20:38.675 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: code, '\n'.join(err))) 2024-01-13T08:20:38.675 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:20:38.676 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpkfbnjqwi:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp04zvn9e0:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:20:38.676 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:20:38.676 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:20:38.676 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:20:38.676 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name cl 2024-01-13T08:20:38.676 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: ient.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 96221318-0687-4c79-a5f1-fc164398f2b6 2024-01-13T08:20:38.676 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:20:38.676 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 --yes-i-really-mean-it 2024-01-13T08:20:38.676 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr stderr: purged osd.1 2024-01-13T08:20:38.676 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:20:38.676 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:20:38.676 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr self.prepare() 2024-01-13T08:20:38.676 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:38.676 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:38.677 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:20:38.677 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:20:38.677 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:20:38.677 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:20:38.677 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:20:38.677 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:20:38.677 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:20:38.677 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:20:38.677 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:20:38.677 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:20:38.677 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:20:38.677 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:20:38.678 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:20:38.678 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:20:38.678 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:20:38.678 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:20:38.678 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:20:38.678 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:20:38.678 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:20:38.678 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:20:38.678 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:20:38.678 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:20:38.679 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:20:38.679 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:20:38.679 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:20:38.679 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:38.679 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:38.679 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:20:38.679 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:20:38.679 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:20:38.679 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:20:38.679 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:38.679 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:38.679 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:20:38.679 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:20:38.679 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:20:38.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:20:38.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:20:38.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:20:38.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:20:38.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:20:38.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:38.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:38.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:20:38.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:20:38.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:20:38.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:20:38.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 1 2024-01-13T08:20:38.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:20:38.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:20:38.681 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: main() 2024-01-13T08:20:38.683 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:20:38.683 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: r = ctx.func(ctx) 2024-01-13T08:20:38.683 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:20:38.683 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:20:38.684 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:20:38.684 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:20:38.684 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:20:38.684 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:20:38.684 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:20:38.684 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:20:38.684 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:20:38.684 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:20:38.684 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:20:38.684 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:20:38.684 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpkfbnjqwi:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp04zvn9e0:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:20:38.685 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:38 smithi131 conmon[38024]: cluster 2024-01-13T08:20:37.272398+0000 mgr.smithi099.oodult (mgr.14180) 121 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:38.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: cephadm 2024-01-13T08:20:36.936187+0000 mgr.smithi099.oodult (mgr.14180) 118 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi131', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:20:38.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:20:38.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:20:38.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: return f(*arg) 2024-01-13T08:20:38.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:20:38.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:20:38.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:20:38.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: code, '\n'.join(err))) 2024-01-13T08:20:38.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:20:38.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpkfbnjqwi:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp04zvn9e0:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:20:38.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:20:38.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:20:38.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:20:38.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 96221318-0687-4c79-a5f1-fc164398f2b6 2024-01-13T08:20:38.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:20:38.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 --yes-i-really-mean-it 2024-01-13T08:20:38.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr stderr: purged osd.1 2024-01-13T08:20:38.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:20:38.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:20:38.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr self.prepare() 2024-01-13T08:20:38.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:38.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:38.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:20:38.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:20:38.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:20:38.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:20:38.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:20:38.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:20:38.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:20:38.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:20:38.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:20:38.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:20:38.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:20:38.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:20:38.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:20:38.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:20:38.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:20:38.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:20:38.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:20:38.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:20:38.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:20:38.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:20:38.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:20:38.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:20:38.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:20:38.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:20:38.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:20:38.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:38.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:38.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:20:38.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:20:38.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:20:38.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:20:38.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:38.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:38.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:20:38.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:20:38.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:20:38.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:20:38.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:20:38.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:20:38.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:20:38.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:20:38.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:38.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:38.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:20:38.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:20:38.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:20:38.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:20:38.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 1 2024-01-13T08:20:38.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:20:38.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:20:38.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: main() 2024-01-13T08:20:38.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:20:38.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: r = ctx.func(ctx) 2024-01-13T08:20:38.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:20:38.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:20:38.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:20:38.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:20:38.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:20:38.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:20:38.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-8777 2024-01-13T08:20:38.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: 4f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:20:38.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:20:38.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:20:38.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:20:38.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:20:38.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:20:38.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpkfbnjqwi:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp04zvn9e0:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:20:38.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: audit 2024-01-13T08:20:37.246251+0000 mon.smithi099 (mon.0) 441 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:20:38.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: cephadm 2024-01-13T08:20:37.270306+0000 mgr.smithi099.oodult (mgr.14180) 119 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi131', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:20:38.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:20:38.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:20:38.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: return f(*arg) 2024-01-13T08:20:38.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:20:38.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:20:38.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:20:38.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: code, '\n'.join(err))) 2024-01-13T08:20:38.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi099/config 2024-01-13T08:20:38.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpkfqs4er9:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmptgqnpszv:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:20:38.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:20:38.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:20:38.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:20:38.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 91656956-5a42-4abe-ab0b-485172d8a222 2024-01-13T08:20:38.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:20:38.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:20:38.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:20:38.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:20:38.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:20:38.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr self.prepare() 2024-01-13T08:20:38.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:38.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:38.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:20:38.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:20:38.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:20:38.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:20:38.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:20:38.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:20:38.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:20:38.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:20:38.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:20:38.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:20:38.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:20:38.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:20:38.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:20:38.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:20:38.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:20:38.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:20:38.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:20:38.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:20:38.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:20:38.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:20:38.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:20:38.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:20:38.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:20:38.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:20:38.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:20:38.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:38.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:38.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:20:38.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:20:38.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:20:38.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:20:38.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:38.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:38.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:20:38.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:20:38.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:20:38.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr r 2024-01-13T08:20:38.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: ollback_osd(self.args, self.osd_id) 2024-01-13T08:20:38.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:20:38.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:20:38.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:20:38.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:20:38.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:38.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:38.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:20:38.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:20:38.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:20:38.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:20:38.746 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:20:38.746 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:20:38.746 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:20:38.746 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: main() 2024-01-13T08:20:38.746 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:20:38.746 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: r = ctx.func(ctx) 2024-01-13T08:20:38.746 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:20:38.746 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:20:38.746 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:20:38.746 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:20:38.746 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:20:38.746 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:20:38.746 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:20:38.746 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:20:38.746 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:20:38.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:20:38.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:20:38.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:20:38.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpkfqs4er9:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmptgqnpszv:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:20:38.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: cephadm 2024-01-13T08:20:37.271742+0000 mgr.smithi099.oodult (mgr.14180) 120 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T08:20:38.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: service_id: all-available-devices 2024-01-13T08:20:38.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: service_name: osd.all-available-devices 2024-01-13T08:20:38.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: placement: 2024-01-13T08:20:38.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: host_pattern: '*' 2024-01-13T08:20:38.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: spec: 2024-01-13T08:20:38.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: data_devices: 2024-01-13T08:20:38.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: all: true 2024-01-13T08:20:38.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: filter_logic: AND 2024-01-13T08:20:38.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: objectstore: bluestore 2024-01-13T08:20:38.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:20:38.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpkfbnjqwi:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp04zvn9e0:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:20:38.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:20:38.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:20:38.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:20:38.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 96221318-0687-4c79-a5f1-fc164398f2b6 2024-01-13T08:20:38.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:20:38.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 --yes-i-really-mean-it 2024-01-13T08:20:38.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr stderr: purged osd.1 2024-01-13T08:20:38.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:20:38.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:20:38.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr self.prepare() 2024-01-13T08:20:38.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:38.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:38.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:20:38.749 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:20:38.749 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:20:38.749 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:20:38.749 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:20:38.749 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:20:38.749 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:20:38.749 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:20:38.749 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:20:38.749 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:20:38.749 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:20:38.749 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:20:38.749 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:20:38.749 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:20:38.749 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:20:38.749 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:20:38.750 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:20:38.750 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:20:38.750 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:20:38.750 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py 2024-01-13T08:20:38.750 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: ", line 194, in dispatch 2024-01-13T08:20:38.750 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:20:38.750 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:20:38.750 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:20:38.750 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:20:38.750 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:20:38.750 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:38.750 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:38.750 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:20:38.750 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:20:38.750 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:20:38.750 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:20:38.751 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:38.751 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:38.751 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:20:38.751 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:20:38.751 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:20:38.751 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:20:38.751 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:20:38.751 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:20:38.751 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:20:38.751 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:20:38.751 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:38.751 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:38.751 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:20:38.751 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:20:38.752 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:20:38.752 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:20:38.752 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 1 2024-01-13T08:20:38.752 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:20:38.752 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:20:38.753 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: main() 2024-01-13T08:20:38.753 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:20:38.753 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: r = ctx.func(ctx) 2024-01-13T08:20:38.753 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:20:38.753 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:20:38.753 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:20:38.753 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:20:38.753 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:20:38.753 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:20:38.753 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:20:38.754 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:20:38.754 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:20:38.754 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:20:38.754 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:20:38.754 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:20:38.754 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpkfbnjqwi:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp04zvn9e0:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:20:38.754 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:20:38.754 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T08:20:38.754 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: if self._apply_service(spec): 2024-01-13T08:20:38.754 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T08:20:38.754 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T08:20:38.754 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T08:20:38.754 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T08:20:38.754 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T08:20:38.754 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T08:20:38.755 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T08:20:38.755 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T08:20:38.755 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T08:20:38.755 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: raise self._value 2024-01-13T08:20:38.755 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T08:20:38.755 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: result = (True, func(*args, **kwds)) 2024-01-13T08:20:38.755 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T08:20:38.755 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: return list(map(*args)) 2024-01-13T08:20:38.755 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:20:38.755 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: return f(*arg) 2024-01-13T08:20:38.755 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:20:38.755 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:20:38.755 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:20:38.755 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: code, '\n'.join(err))) 2024-01-13T08:20:38.755 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:20:38.756 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpkfbnjqwi:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp04zvn9e0:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:20:38.756 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:20:38.756 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:20:38.756 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:20:38.756 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name cl 2024-01-13T08:20:38.756 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: ient.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 96221318-0687-4c79-a5f1-fc164398f2b6 2024-01-13T08:20:38.756 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:20:38.756 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 --yes-i-really-mean-it 2024-01-13T08:20:38.756 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr stderr: purged osd.1 2024-01-13T08:20:38.756 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:20:38.756 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:20:38.756 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr self.prepare() 2024-01-13T08:20:38.756 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:38.756 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:38.757 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:20:38.757 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:20:38.757 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:20:38.757 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:20:38.757 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:20:38.757 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:20:38.757 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:20:38.757 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:20:38.757 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:20:38.757 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:20:38.757 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:20:38.757 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:20:38.757 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:20:38.757 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:20:38.757 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:20:38.758 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:20:38.758 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:20:38.758 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:20:38.758 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:20:38.758 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:20:38.758 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:20:38.758 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:20:38.758 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:20:38.758 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:20:38.758 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:20:38.758 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:38.758 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:38.758 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:20:38.758 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:20:38.758 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:20:38.759 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:20:38.759 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:38.759 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:38.759 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:20:38.759 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:20:38.759 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:20:38.759 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:20:38.759 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:20:38.759 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:20:38.759 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:20:38.759 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:20:38.759 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:20:38.759 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:20:38.759 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:20:38.759 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:20:38.760 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:20:38.760 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:20:38.760 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 1 2024-01-13T08:20:38.760 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:20:38.760 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:20:38.760 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: main() 2024-01-13T08:20:38.760 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:20:38.760 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: r = ctx.func(ctx) 2024-01-13T08:20:38.760 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:20:38.760 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:20:38.760 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:20:38.760 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:20:38.760 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:20:38.760 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:20:38.760 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:20:38.761 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:20:38.761 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:20:38.761 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:20:38.761 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:20:38.761 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:20:38.761 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpkfbnjqwi:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp04zvn9e0:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:20:38.761 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:38 smithi099 conmon[33313]: cluster 2024-01-13T08:20:37.272398+0000 mgr.smithi099.oodult (mgr.14180) 121 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:39.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:39 smithi099 conmon[33313]: cluster 2024-01-13T08:20:38.244572+0000 mon.smithi099 (mon.0 2024-01-13T08:20:39.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:39 smithi099 conmon[33313]: ) 442 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T08:20:39.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:39 smithi131 conmon[38024]: cluster 2024-01-13T08:20:38.244572+0000 mon.smithi099 (mon.0) 442 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T08:20:39.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:39 smithi131 conmon[38024]: 2024-01-13T08:20:40.133 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:20:40.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:40 smithi099 conmon[33313]: cluster 2024-01-13T08:20:39.272795+0000 mgr.smithi099.oodult (mgr.14180) 122 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:40.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:40 smithi099 conmon[33313]: audit 2024-01-13T08:20:40 2024-01-13T08:20:40.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:40 smithi099 conmon[33313]: .137255+0000 mon.smithi099 (mon.0) 443 : audit [DBG] from='client.? 172.21.15.99:0/3628202391' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:20:40.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:40 smithi131 conmon[38024]: cluster 2024-01-13T08:20:39.272795+0000 mgr.smithi099.oodult (mgr.14180) 122 : cluster 2024-01-13T08:20:40.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:40 smithi131 conmon[38024]: [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:40.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:40 smithi131 conmon[38024]: audit 2024-01-13T08:20:40.137255+0000 mon.smithi099 (mon.0) 443 : audit [DBG] from='client.? 172.21.15.99:0/3628202391' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:20:40.748 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:20:41.750 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:20:42.207 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:41 smithi099 conmon[33313]: cluster 2024-01-13T08:20:41.273232+0000 mgr.smithi099.oodult (mgr.14180) 123 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:42.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:41 smithi131 conmon[38024]: cluster 2024-01-13T08:20:41.273232+0000 mgr.smithi099.oodult (mgr.14180) 123 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:42.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:41 smithi131 conmon[38024]: 2024-01-13T08:20:43.557 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:20:44.096 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:20:44.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:44 smithi099 conmon[33313]: cluster 2024-01-13T08:20:43.273616+0000 mgr.smithi099.oodult (mgr.14180) 124 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:44.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:44 smithi099 conmon[33313]: audit 2024-01-13T08 2024-01-13T08:20:44.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:44 smithi099 conmon[33313]: :20:43.560915+0000 mon.smithi099 (mon.0) 444 : audit [DBG] from='client.? 172.21.15.99:0/2810831509' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:20:44.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:44 smithi131 conmon[38024]: cluster 2024-01-13T08:20:43.273616+0000 mgr.smithi099.oodult (mgr.14180) 124 : cluster 2024-01-13T08:20:44.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:44 smithi131 conmon[38024]: [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:44.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:44 smithi131 conmon[38024]: audit 2024-01-13T08:20:43.560915+0000 mon.smithi099 (mon.0) 444 : audit [DBG] 2024-01-13T08:20:44.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:44 smithi131 conmon[38024]: from='client.? 172.21.15.99:0/2810831509' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:20:45.098 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:20:46.537 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:46 smithi099 conmon[33313]: cluster 2024-01-13T08:20:45.273982+0000 mgr.smithi099.oodult (mgr.14180 2024-01-13T08:20:46.537 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:46 smithi099 conmon[33313]: ) 125 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:46.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:46 smithi131 conmon[38024]: cluster 2024-01-13T08:20:45.273982+0000 mgr.smithi099.oodult (mgr.14180) 125 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:46.872 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:20:47.471 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:20:47.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:47 smithi131 conmon[38024]: audit 2024-01-13T08:20:46.876569+0000 mon.smithi099 (mon.0) 445 : audit [DBG] from='client.? 172.21.15.99:0/542281207' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:20:47.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:47 smithi099 conmon[33313]: audit 2024-01-13T08:20:46.876569+0000 mon.smithi099 (mon.0) 445 : audit [DBG] 2024-01-13T08:20:47.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:47 smithi099 conmon[33313]: from='client.? 172.21.15.99:0/542281207' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:20:48.473 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:20:48.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:48 smithi131 conmon[38024]: cluster 2024-01-13T08:20:47.274477+0000 mgr.smithi099.oodult (mgr.14180) 126 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:48.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:48 smithi099 conmon[33313]: cluster 2024-01-13T08:20:47.274477+0000 mgr.smithi099.oodult (mgr.14180) 126 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:48.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:48 smithi099 conmon[33313]: 2024-01-13T08:20:50.312 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:20:50.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:50 smithi131 conmon[38024]: cluster 2024-01-13T08:20:49.274858+0000 mgr.smithi099.oodult (mgr.14180) 127 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:50.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:50 smithi131 conmon[38024]: audit 2024-01-13T08:20:50.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:50 smithi131 conmon[38024]: 2024-01-13T08:20:50.315705+0000 mon.smithi099 (mon.0) 446 : audit [DBG] from='client.? 172.21.15.99:0/2382673234' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:20:50.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:50 smithi099 conmon[33313]: cluster 2024-01-13T08:20:49.274858+0000 mgr.smithi099.oodult (mgr.14180) 127 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:50.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:50 smithi099 conmon[33313]: audit 2024-01-13T08:20:50.315705 2024-01-13T08:20:50.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:50 smithi099 conmon[33313]: +0000 mon.smithi099 (mon.0) 446 : audit [DBG] from='client.? 172.21.15.99:0/2382673234' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:20:50.888 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:20:51.889 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:20:52.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:51 smithi099 conmon[33313]: cluster 2024-01-13T08:20:51.275307+0000 mgr.smithi099.oodult (mgr.14180) 128 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:52.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:51 smithi131 conmon[38024]: cluster 2024-01-13T08:20:51.275307+0000 mgr.smithi099.oodult (mgr.14180) 128 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:53.313 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:53 smithi099 conmon[33313]: audit 2024-01-13T08:20:52.246241+0000 mon.smithi099 (mon.0) 2024-01-13T08:20:53.313 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:53 smithi099 conmon[33313]: 447 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:20:53.626 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:20:53.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:53 smithi131 conmon[38024]: audit 2024-01-13T08:20:52.246241+0000 mon.smithi099 (mon.0) 447 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:20:54.188 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:20:54.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:54 smithi099 conmon[33313]: cluster 2024-01-13T08:20:53.275663+0000 mgr.smithi099.oodult (mgr. 2024-01-13T08:20:54.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:54 smithi099 conmon[33313]: 14180) 129 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:54.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:54 smithi099 conmon[33313]: audit 2024-01-13T08:20:53.630137+0000 mon.smithi099 (mon.0) 448 : audit [DBG] from='client.? 172.21.15.99:0/2225519468' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:20:54.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:54 smithi131 conmon[38024]: cluster 2024-01-13T08:20:53.275663+0000 mgr.smithi099.oodult (mgr.14180) 129 : cluster [DBG] 2024-01-13T08:20:54.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:54 smithi131 conmon[38024]: pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:54.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:54 smithi131 conmon[38024]: audit 2024-01-13T08:20:53.630137+0000 mon.smithi099 (mon.0) 448 : audit [DBG] from='client.? 172.21.15.99:0/2225519468' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:20:55.189 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:20:56.597 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:56 smithi099 conmon[33313]: cluster 2024-01-13T08:20:55.276095+0000 mgr.smithi099.oodult (mgr.14180) 130 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:56.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:56 smithi131 conmon[38024]: cluster 2024-01-13T08:20:55.276095+0000 mgr.smithi099.oodult (mgr.14180) 130 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:56.960 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:20:57.561 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:20:57.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:57 smithi131 conmon[38024]: audit 2024-01-13T08:20:56.964135+0000 mon.smithi099 (mon.0) 449 : audit [DBG] from='client.? 172.21.15.99:0/941458139' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:20:57.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:57 smithi099 conmon[33313]: audit 2024-01-13T08:20:56.964135+0000 mon.smithi099 (mon 2024-01-13T08:20:57.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:57 smithi099 conmon[33313]: .0) 449 : audit [DBG] from='client.? 172.21.15.99:0/941458139' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:20:58.562 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:20:58.595 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:20:58 smithi099 conmon[33313]: cluster 2024-01-13T08:20:57.276648+0000 mgr.smithi099.oodult (mgr.14180) 131 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:20:58.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:58 smithi131 conmon[38024]: cluster 2024-01-13T08:20:57.276648+0000 mgr.smithi099.oodult (mgr.14180) 131 : cluster [DBG] 2024-01-13T08:20:58.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:20:58 smithi131 conmon[38024]: pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:00.308 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:21:00.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:00 smithi131 conmon[38024]: cluster 2024-01-13T08:20:59.277284+0000 mgr.smithi099.oodult (mgr.14180) 132 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:00.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:00 smithi131 conmon[38024]: audit 2024-01-13T08:21:00.311789+0000 mon.smithi099 (mon.0) 450 : audit [DBG] from='client.? 172.21.15.99:0/2740170198' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:21:00.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:00 smithi099 conmon[33313]: cluster 2024-01-13T08:20:59.277284+0000 mgr.smithi099.oodult (mgr.14180) 132 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:00.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:00 smithi099 conmon[33313]: audit 2024- 2024-01-13T08:21:00.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:00 smithi099 conmon[33313]: 01-13T08:21:00.311789+0000 mon.smithi099 (mon.0) 450 : audit [DBG] from='client.? 172.21.15.99:0/2740170198' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:21:00.861 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:21:01.862 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:21:02.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:01 smithi099 conmon[33313]: cluster 2024-01-13T08:21:01.277683+0000 mgr.smithi099.oodult (mgr.14180) 133 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:02.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:01 smithi131 conmon[38024]: cluster 2024-01-13T08:21:01.277683+0000 mgr.smithi099.oodult (mgr.14180) 133 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:03.618 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:21:04.200 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:21:04.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:04 smithi099 conmon[33313]: cluster 2024-01-13T08:21:03.278073+0000 mgr.smithi099.oodult (mgr.14180) 134 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:04.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:04 smithi099 conmon[33313]: audit 2024-01-13T08:21:03.622417+0000 mon.smithi099 (mon.0) 451 : audit [DBG] from='client.? 172.21.15.99:0/503959279' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:21:04.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:04 smithi131 conmon[38024]: cluster 2024-01-13T08:21:03.278073+0000 mgr.smithi099.oodult (mgr.14180) 134 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:04.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:04 smithi131 conmon[38024]: 2024-01-13T08:21:04.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:04 smithi131 conmon[38024]: audit 2024-01-13T08:21:03.622417+0000 mon.smithi099 (mon.0) 451 : audit [DBG] from='client.? 172.21.15.99:0/503959279' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:21:05.202 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:21:06.582 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:06 smithi099 conmon[33313]: cluster 2024-01-13T08:21:05.278505 2024-01-13T08:21:06.582 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:06 smithi099 conmon[33313]: +0000 mgr.smithi099.oodult (mgr.14180) 135 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:06.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:06 smithi131 conmon[38024]: cluster 2024-01-13T08:21:05.278505+0000 mgr.smithi099.oodult (mgr.14180) 135 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:06.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:06 smithi131 conmon[38024]: 2024-01-13T08:21:06.886 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:21:07.433 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:21:07.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:07 smithi131 conmon[38024]: audit 2024-01-13T08:21:06.890126+0000 mon.smithi099 (mon.0) 452 : audit [DBG] from='client.? 172.21.15.99:0/2819308153' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:21:07.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:07 smithi131 conmon[38024]: 2024-01-13T08:21:07.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:07 smithi099 conmon[33313]: audit 2024-01-13T08:21:06.890126+0000 mon.smithi099 (mon.0) 452 : audit 2024-01-13T08:21:07.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:07 smithi099 conmon[33313]: [DBG] from='client.? 172.21.15.99:0/2819308153' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:21:08.435 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:21:08.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:08 smithi131 conmon[38024]: cluster 2024-01-13T08:21:07.279291+0000 mgr.smithi099.oodult (mgr.14180) 136 2024-01-13T08:21:08.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:08 smithi131 conmon[38024]: : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:08.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:08 smithi099 conmon[33313]: cluster 2024-01-13T08:21:07.279291+0000 mgr.smithi099.oodult (mgr.14180) 136 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:10.203 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:21:10.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:10 smithi099 conmon[33313]: cluster 2024-01-13T08:21:09.279800+0000 mgr.smithi099.oodult (mgr.14180) 137 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:10.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:10 smithi099 conmon[33313]: audit 2024-01-13T08:21:10.205783 2024-01-13T08:21:10.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:10 smithi099 conmon[33313]: +0000 mon.smithi099 (mon.0) 453 : audit [DBG] from='client.? 172.21.15.99:0/2079726795' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:21:10.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:10 smithi131 conmon[38024]: cluster 2024-01-13T08:21:09.279800+0000 mgr.smithi099.oodult (mgr.14180) 137 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:10.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:10 smithi131 conmon[38024]: audit 2024-01-13T08:21:10.205783+0000 mon.smithi099 (mon.0) 453 : audit [DBG] from='client.? 172.21.15.99:0/2079726795' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:21:10.969 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:21:11.971 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:21:12.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:11 smithi131 conmon[38024]: cluster 2024-01-13T08:21:11.280241+0000 mgr.smithi099.oodult (mgr.14180) 138 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:12.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:11 smithi131 conmon[38024]: 2024-01-13T08:21:12.416 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:11 smithi099 conmon[33313]: cluster 2024-01-13T08:21:11.280241+0000 mgr.smithi099.oodult (mgr.14180) 138 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:12.416 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:11 smithi099 conmon[33313]: 2024-01-13T08:21:13.797 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:21:14.349 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:21:14.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:14 smithi131 conmon[38024]: cluster 2024-01-13T08:21:13.280620+0000 mgr.smithi099.oodult (mgr.14180) 139 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:14.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:14 smithi131 conmon[38024]: audit 2024-01-13T08:21:13.801096+0000 mon.smithi099 (mon.0) 454 : audit [DBG] from='client.? 172.21.15.99:0/3175019083' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:21:14.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:14 smithi099 conmon[33313]: cluster 2024-01-13T08:21:13.280620+0000 mgr.smithi099.oodult (mgr.14180) 139 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:14.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:14 smithi099 conmon[33313]: audit 2024-01-13T08:21:13.801096+0000 mon.smithi099 (mon.0) 454 : audit [DBG] from='client.? 172.21.15.99:0/3175019083' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:21:15.351 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:21:16.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:16 smithi099 conmon[33313]: cluster 2024-01-13T08:21:15.281051+0000 mgr.smithi099.oodult (mgr.14180) 140 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:16.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:16 smithi131 conmon[38024]: cluster 2024-01-13T08:21:15.281051+0000 mgr.smithi099.oodult (mgr.14180) 140 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:17.126 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:21:17.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:17 smithi099 conmon[33313]: audit 2024-01-13T08:21:17.129512+0000 mon.smithi099 (mon.0) 455 : audit [DBG] 2024-01-13T08:21:17.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:17 smithi099 conmon[33313]: from='client.? 172.21.15.99:0/1620863788' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:21:17.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:17 smithi131 conmon[38024]: audit 2024-01-13T08:21:17.129512+0000 mon.smithi099 (mon.0) 455 : audit [DBG] from='client.? 172.21.15.99:0/1620863788' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:21:17.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:17 smithi131 conmon[38024]: 2024-01-13T08:21:17.689 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:21:18.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:18 smithi131 conmon[38024]: cluster 2024-01-13T08:21:17.281740+0000 mgr.smithi099.oodult (mgr.14180) 141 : cluster 2024-01-13T08:21:18.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:18 smithi131 conmon[38024]: [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:18.691 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:21:18.724 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:18 smithi099 conmon[33313]: cluster 2024-01-13T08:21:17.281740+0000 mgr.smithi099.oodult (mgr.14180) 141 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:20.469 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:21:20.640 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:20 smithi099 conmon[33313]: cluster 2024-01-13T08:21:19.282171+0000 mgr.smithi099.oodult (mgr.14180) 142 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:20.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:20 smithi131 conmon[38024]: cluster 2024-01-13T08:21:19.282171+0000 mgr.smithi099.oodult (mgr.14180) 142 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:21.030 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:21:21.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:21 smithi131 conmon[38024]: audit 2024-01-13T08:21:20.472568+0000 mon.smithi099 (mon.0) 456 : audit [DBG] from='client.? 172.21.15.99:0/1357059215' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:21:21.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:21 smithi099 conmon[33313]: audit 2024-01-13T08:21:20.472568+0000 mon.smithi099 (mon.0) 456 : audit [DBG] from='client.? 172.21.15.99:0/1357059215' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:21:22.031 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:21:22.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:22 smithi131 conmon[38024]: cluster 2024-01-13T08:21:21.282854+0000 mgr.smithi099.oodult (mgr.14180) 143 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:22.674 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:22 smithi099 conmon[33313]: cluster 2024-01-13T08:21:21.282854+0000 mgr.smithi099.oodult (mgr.14180) 143 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:23.813 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:21:24.347 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:21:24.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:24 smithi131 conmon[38024]: cluster 2024-01-13T08:21:23.283287+0000 mgr.smithi099.oodult (mgr.14180) 144 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:24.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:24 smithi131 conmon[38024]: audit 2024-01-13T08:21:23.817076+0000 mon.smithi099 (mon.0) 457 : audit [DBG] from='client.? 172.21.15.99:0/2569851447' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:21:24.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:24 smithi099 conmon[33313]: cluster 2024-01-13T08:21:23.283287+0000 mgr.smithi099.oodult (mgr.14180) 144 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:24.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:24 smithi099 conmon[33313]: audit 2024-01-13T08:21:23.817076+0000 mon.smithi099 (mon.0) 457 : audit [DBG] from='client.? 172.21.15.99:0/2569851447' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:21:25.349 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:21:26.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:26 smithi099 conmon[33313]: cluster 2024-01-13T08:21:25.283713+0000 mgr.smithi099.oodult (mgr.14180) 145 : cluster 2024-01-13T08:21:26.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:26 smithi099 conmon[33313]: [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:26.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:26 smithi131 conmon[38024]: cluster 2024-01-13T08:21:25.283713+0000 mgr.smithi099.oodult (mgr.14180) 145 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:27.081 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:21:27.648 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:27 smithi099 conmon[33313]: audit 2024-01-13T08:21:27.084804+0000 mon.smithi099 (mon.0) 458 : audit [DBG] from='client.? 172.21.15.99:0/3892464393' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:21:27.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:27 smithi131 conmon[38024]: audit 2024-01-13T08:21:27.084804+0000 mon.smithi099 (mon.0) 458 : audit [DBG] from='client.? 172.21.15.99:0/3892464393' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:21:27.663 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:21:28.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:28 smithi131 conmon[38024]: cluster 2024-01-13T08:21:27.284338+0000 mgr.smithi099.oodult (mgr.14180) 146 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:28.665 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:21:28.698 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:28 smithi099 conmon[33313]: cluster 2024-01-13T08:21:27.284338+0000 mgr.smithi099.oodult (mgr.14180) 146 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:30.458 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:21:30.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:30 smithi131 conmon[38024]: cluster 2024-01-13T08:21:29.284677+0000 mgr.smithi099.oodult (mgr.14180) 147 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:30.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:30 smithi099 conmon[33313]: cluster 2024-01-13T08:21:29.284677+0000 mgr.smithi099.oodult (mgr.14180) 2024-01-13T08:21:30.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:30 smithi099 conmon[33313]: 147 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:31.006 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:21:31.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:31 smithi131 conmon[38024]: audit 2024-01-13T08:21:30.462068+0000 mon.smithi099 (mon.0) 459 : audit [DBG] 2024-01-13T08:21:31.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:31 smithi131 conmon[38024]: from='client.? 172.21.15.99:0/3009349975' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:21:31.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:31 smithi099 conmon[33313]: audit 2024-01-13T08:21:30.462068+0000 mon.smithi099 (mon.0) 459 : audit [DBG] from='client.? 172.21.15.99:0/3009349975' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:21:32.007 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:21:32.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:32 smithi131 conmon[38024]: cluster 2024-01-13T08:21:31.285366+0000 mgr.smithi099.oodult (mgr.14180) 148 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:32.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:32 smithi099 conmon[33313]: cluster 2024-01-13T08:21:31.285366+0000 mgr.smithi099.oodult (mgr.14180 2024-01-13T08:21:32.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:32 smithi099 conmon[33313]: ) 148 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:33.756 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:21:34.294 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:21:34.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:34 smithi131 conmon[38024]: cluster 2024-01-13T08:21:33.285746+0000 mgr.smithi099.oodult (mgr.14180) 149 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:34.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:34 smithi131 conmon[38024]: audit 2024-01-13T08:21:33.759981+0000 mon.smithi099 (mon.0) 460 : audit [DBG] from='client.? 172.21.15.99:0/1281243522' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:21:34.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:34 smithi099 conmon[33313]: cluster 2024-01-13T08:21:33.285746+0000 mgr.smithi099.oodult (mgr.14180) 149 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:34.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:34 smithi099 conmon[33313]: audit 2024-01-13T08:21:33.759981+0000 mon.smithi099 (mon.0) 460 : audit [DBG] from='client.? 172.21.15.99:0/1281243522' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:21:35.296 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:21:36.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:36 smithi131 conmon[38024]: cluster 2024-01-13T08:21:35.286216+0000 mgr.smithi099.oodult (mgr.14180) 150 : cluster 2024-01-13T08:21:36.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:36 smithi131 conmon[38024]: [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:36.687 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:36 smithi099 conmon[33313]: cluster 2024-01-13T08:21:35.286216+0000 mgr.smithi099.oodult (mgr.14180) 150 2024-01-13T08:21:36.687 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:36 smithi099 conmon[33313]: : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:37.042 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:21:37.619 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":17,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:21:37.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:37 smithi131 conmon[38024]: audit 2024-01-13T08:21:37.045840+0000 mon.smithi099 (mon.0) 461 : audit [DBG] from='client.? 172.21.15.99:0/2600924317' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:21:37.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:37 smithi131 conmon[38024]: audit 2024-01-13T08:21:37.273833+0000 mon.smithi099 (mon.0) 462 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:21:37.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:37 smithi099 conmon[33313]: audit 2024-01-13T08:21:37.045840+0000 mon.smithi099 (mon.0) 461 : audit [DBG] from='client.? 172.21.15.99:0/2600924317' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:21:37.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:37 smithi099 conmon[33313]: audit 2024-01-13T08:21:37.273833+0000 mon.smithi099 (mon.0) 462 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:21:38.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:38 smithi099 conmon[33313]: cluster 2024-01-13T08:21:37.286619+0000 mgr.smithi099.oodult (mgr. 2024-01-13T08:21:38.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:38 smithi099 conmon[33313]: 14180) 151 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:38.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:38 smithi099 conmon[33313]: audit 2024-01-13T08:21:37.675545+0000 mon.smithi099 (mon.0) 463 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:21:38.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:38 smithi099 conmon[33313]: audit 2024-01-13T08:21:37.676149+0000 mon.smithi099 (mon.0) 464 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T08:21:38.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:38 smithi099 conmon[33313]: audit 2024-01-13T08:21:37.680100+0000 mon.smithi099 (mon.0) 465 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:21:38.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:38 smithi099 conmon[33313]: cluster 2024-01-13T08:21:37.680385+0000 mgr.smithi099.oodult (mgr.14180) 152 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:38.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:38 smithi099 conmon[33313]: audit 2024-01-13T08:21:37.683359+0000 mon.smithi099 (mon.0) 466 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T08:21:38.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:38 smithi099 conmon[33313]: audit 2024-01-13T08:21:37.685949+0000 mon.smithi099 (mon.0) 467 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:21:38.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:38 smithi099 conmon[33313]: audit 2024-01-13T08:21:37.686804+0000 mon.smithi099 (mon.0) 468 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:21:38.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:38 smithi099 conmon[33313]: audit 2024-01-13T08:21:37.687052+0000 mon.smithi099 (mon.0) 469 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:21:38.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:38 smithi099 conmon[33313]: audit 2024-01-13T08:21:37.687463+0000 mon.smithi099 (mon.0) 470 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:21:38.574 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:38 smithi131 conmon[38024]: cluster 2024-01-13T08:21:37.286619+0000 2024-01-13T08:21:38.575 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:38 smithi131 conmon[38024]: mgr.smithi099.oodult (mgr.14180) 151 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:38.575 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:38 smithi131 conmon[38024]: audit 2024-01-13T08:21:37.675545+0000 mon.smithi099 (mon.0) 463 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:21:38.575 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:38 smithi131 conmon[38024]: audit 2024-01-13T08:21:37.676149+0000 mon.smithi099 (mon.0) 464 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T08:21:38.575 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:38 smithi131 conmon[38024]: audit 2024-01-13T08:21:37.680100+0000 mon.smithi099 (mon.0) 465 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:21:38.575 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:38 smithi131 conmon[38024]: cluster 2024-01-13T08:21:37.680385+0000 mgr.smithi099.oodult (mgr.14180) 152 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:38.575 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:38 smithi131 conmon[38024]: audit 2024-01-13T08:21:37.683359+0000 mon.smithi099 (mon.0) 466 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T08:21:38.575 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:38 smithi131 conmon[38024]: audit 2024-01-13T08:21:37.685949+0000 mon.smithi099 (mon.0) 467 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:21:38.575 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:38 smithi131 conmon[38024]: audit 2024-01-13T08:21:37.686804+0000 mon.smithi099 (mon.0) 468 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:21:38.575 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:38 smithi131 conmon[38024]: audit 2024-01-13T08:21:37.687052+0000 mon.smithi099 (mon.0) 469 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:21:38.575 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:38 smithi131 conmon[38024]: audit 2024-01-13T08:21:37.687463+0000 mon.smithi099 (mon.0) 470 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:21:38.620 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:21:39.417 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:39 smithi099 conmon[33313]: cluster 2024-01-13T08:21:38.678545+0000 mon.smithi099 (mon.0) 471 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T08:21:39.417 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:39 smithi099 conmon[33313]: cluster 2024-01-13T08:21:38.678600+0000 mon.smithi099 (mon.0) 472 : cluster [INF] Cluster is now healthy 2024-01-13T08:21:39.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:39 smithi131 conmon[38024]: cluster 2024-01-13T08:21:38.678545+0000 mon.smithi099 (mon.0) 471 : cluster [INF] 2024-01-13T08:21:39.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:39 smithi131 conmon[38024]: Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T08:21:39.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:39 smithi131 conmon[38024]: cluster 2024-01-13T08:21:38.678600+0000 mon.smithi099 (mon.0) 472 : cluster [INF] Cluster is now healthy 2024-01-13T08:21:40.671 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:40 smithi131 conmon[38024]: cluster 2024-01-13T08:21:39.680612+0000 mgr.smithi099.oodult (mgr.14180) 153 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:40.671 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:40 smithi131 conmon[38024]: audit 2024-01-13T08:21:40.285705+0000 mon.smithi131 (mon.1) 8 : audit [INF] from='client.? 172.21.15.131:0/611310282' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a228472f-72cf-434d-9146-7bb523644b3c"}]: dispatch 2024-01-13T08:21:40.671 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:40 smithi131 conmon[38024]: audit 2024-01-13T08:21:40.286068+0000 mon.smithi099 (mon.0) 473 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a228472f-72cf-434d-9146-7bb523644b3c"}]: dispatch 2024-01-13T08:21:40.671 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:40 smithi131 conmon[38024]: audit 2024-01-13T08:21:40.289356+0000 mon.smithi099 (mon.0) 474 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a228472f-72cf-434d-9146-7bb523644b3c"}]': finished 2024-01-13T08:21:40.672 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:40 smithi131 conmon[38024]: cluster 2024-01-13T08:21:40.289412+0000 mon.smithi099 (mon.0) 475 : cluster [DBG] osdmap e18: 1 total, 0 up, 1 in 2024-01-13T08:21:40.672 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:40 smithi131 conmon[38024]: audit 2024-01-13T08:21:40.289504+0000 mon.smithi099 (mon.0) 476 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:21:40.704 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:21:40.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:40 smithi099 conmon[33313]: cluster 2024-01-13T08:21:39.680612+0000 mgr.smithi099.oodult (mgr. 2024-01-13T08:21:40.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:40 smithi099 conmon[33313]: 14180) 153 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:40.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:40 smithi099 conmon[33313]: audit 2024-01-13T08:21:40.285705+0000 mon.smithi131 (mon.1) 8 : audit [INF] from='client.? 172.21.15.131:0/611310282' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a228472f-72cf-434d-9146-7bb523644b3c"}]: dispatch 2024-01-13T08:21:40.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:40 smithi099 conmon[33313]: audit 2024-01-13T08:21:40.286068+0000 mon.smithi099 (mon.0) 473 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a228472f-72cf-434d-9146-7bb523644b3c"}]: dispatch 2024-01-13T08:21:40.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:40 smithi099 conmon[33313]: audit 2024-01-13T08:21:40.289356+0000 mon.smithi099 (mon.0) 474 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a228472f-72cf-434d-9146-7bb523644b3c"}]': finished 2024-01-13T08:21:40.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:40 smithi099 conmon[33313]: cluster 2024-01-13T08:21:40.289412+0000 mon.smithi099 (mon.0) 475 : cluster [DBG] osdmap e18: 1 total, 0 up, 1 in 2024-01-13T08:21:40.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:40 smithi099 conmon[33313]: audit 2024-01-13T08:21:40.289504+0000 mon.smithi099 (mon.0) 476 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:21:41.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: audit 2024-01-13T08:21:40.475269+0000 mon.smithi099 (mon.0) 477 : audit [INF] from='client.? 172.21.15.99:0/1097424746' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "25534e84-f268-442f-83bf-5bb185537503"}]: dispatch 2024-01-13T08:21:41.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: 2024-01-13T08:21:41.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: audit 2024-01-13T08:21:40.478506+0000 mon.smithi099 (mon.0) 478 : audit [INF] from='client.? 172.21.15.99:0/1097424746' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "25534e84-f268-442f-83bf-5bb185537503"}]': finished 2024-01-13T08:21:41.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: cluster 2024-01-13T08: 2024-01-13T08:21:41.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: 21:40.478559+0000 mon.smithi099 (mon.0) 479 : cluster [DBG] osdmap e19: 2 total, 0 up, 2 in 2024-01-13T08:21:41.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: audit 2024-01- 2024-01-13T08:21:41.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: 13T08:21:40.478637+0000 mon.smithi099 (mon.0) 480 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:21:41.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: audit 2024-01-13T08:21 2024-01-13T08:21:41.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: :40.478877+0000 mon.smithi099 (mon.0) 481 : audit [DBG] 2024-01-13T08:21:41.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-13T08:21:41.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: audit 2024-01-13T08:21:41.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: 2024-01-13T08:21:40.580433+0000 mon.smithi131 (mon. 2024-01-13T08:21:41.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: 1) 9 2024-01-13T08:21:41.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: : audit [INF] from='client.? 172.21.15.131:0/1847233241' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:21:41.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: audit 2024- 2024-01-13T08:21:41.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: 01-13T08:21:40.580777+0000 mon.smithi099 2024-01-13T08:21:41.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: (mon.0) 482 : audit 2024-01-13T08:21:41.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:21:41.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: audit 2024-01-13T08:21: 2024-01-13T08:21:41.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: 40.585345+0000 mon.smithi099 (mon.0) 483 : audit 2024-01-13T08:21:41.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:21:41.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: cluster 2024-01-13T08:21:40. 2024-01-13T08:21:41.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: 585421+0000 mon.smithi099 (mon.0) 2024-01-13T08:21:41.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: 484 : cluster [DBG] osdmap e20: 1 total, 0 up, 1 in 2024-01-13T08:21:41.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: audit 2024-01-13T08: 2024-01-13T08:21:41.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: 21:40.585522+0000 mon.smithi099 (mon.0) 485 2024-01-13T08:21:41.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-13T08:21:41.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: audit 2024-01-13T08:21: 2024-01-13T08:21:41.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: 40.708194+0000 mon.smithi099 (mon.0) 486 : audit 2024-01-13T08:21:41.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: [DBG] from='client.? 172.21.15.99:0/3989119255' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:21:41.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: audit 2024-01-13T08:21:41.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: 2024-01-13T08 2024-01-13T08:21:41.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: :21:40.782823+0000 mon.smithi099 (mon.0 2024-01-13T08:21:41.485 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: ) 487 : audit [INF] from='client.? 172.21.15.99:0/4009907438' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:21:41.485 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: audit 2024-01-13T08 2024-01-13T08:21:41.485 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: :21:40.786881+0000 mon.smithi099 (mon.0) 2024-01-13T08:21:41.485 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: 488 : audit [INF] from='client.? 172.21.15.99:0/4009907438' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:21:41.485 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: cluster 2024-01-13T08:21 2024-01-13T08:21:41.485 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: :40.786925+0000 mon.smithi099 (mon.0) 489 : cluster 2024-01-13T08:21:41.485 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:41 smithi099 conmon[33313]: [DBG] osdmap e21: 0 total, 0 up, 0 in 2024-01-13T08:21:41.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:41 smithi131 conmon[38024]: audit 2024-01-13T08:21:40.475269+0000 mon.smithi099 (mon.0) 477 : audit [INF] 2024-01-13T08:21:41.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:41 smithi131 conmon[38024]: from='client.? 172.21.15.99:0/1097424746' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "25534e84-f268-442f-83bf-5bb185537503"}]: dispatch 2024-01-13T08:21:41.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:41 smithi131 conmon[38024]: audit 2024-01-13T08:21:40.478506+0000 mon.smithi099 (mon.0) 478 : audit [INF] from='client.? 172.21.15.99:0/1097424746' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "25534e84-f268-442f-83bf-5bb185537503"}]': finished 2024-01-13T08:21:41.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:41 smithi131 conmon[38024]: cluster 2024-01-13T08:21:40.478559+0000 mon.smithi099 (mon.0) 479 : cluster [DBG] osdmap e19: 2 total, 0 up, 2 in 2024-01-13T08:21:41.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:41 smithi131 conmon[38024]: audit 2024-01-13T08:21:40.478637+0000 mon.smithi099 (mon.0) 480 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:21:41.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:41 smithi131 conmon[38024]: audit 2024-01-13T08:21:40.478877+0000 mon.smithi099 (mon.0) 481 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-13T08:21:41.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:41 smithi131 conmon[38024]: audit 2024-01-13T08:21:40.580433+0000 mon.smithi131 (mon.1) 9 : audit [INF] from='client.? 172.21.15.131:0/1847233241' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:21:41.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:41 smithi131 conmon[38024]: audit 2024-01-13T08:21:40.580777+0000 mon.smithi099 (mon.0) 482 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:21:41.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:41 smithi131 conmon[38024]: audit 2024-01-13T08:21:40.585345+0000 mon.smithi099 (mon.0) 483 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:21:41.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:41 smithi131 conmon[38024]: cluster 2024-01-13T08:21:40.585421+0000 mon.smithi099 (mon.0) 484 : cluster [DBG] osdmap e20: 1 total, 0 up, 1 in 2024-01-13T08:21:41.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:41 smithi131 conmon[38024]: audit 2024-01-13T08:21:40.585522+0000 mon.smithi099 (mon.0) 485 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-13T08:21:41.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:41 smithi131 conmon[38024]: audit 2024-01-13T08:21:40.708194+0000 mon.smithi099 (mon.0) 486 : audit [DBG] from='client.? 172.21.15.99:0/3989119255' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:21:41.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:41 smithi131 conmon[38024]: audit 2024-01-13T08:21:40.782823+0000 mon.smithi099 (mon.0) 487 : audit [INF] from='client.? 172.21.15.99:0/4009907438' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:21:41.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:41 smithi131 conmon[38024]: audit 2024-01-13T08:21:40.786881+0000 mon.smithi099 (mon.0) 488 : audit [INF] from='client.? 172.21.15.99:0/4009907438' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:21:41.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:41 smithi131 conmon[38024]: cluster 2024-01-13T08:21:40.786925+0000 mon.smithi099 (mon.0) 489 : cluster [DBG] osdmap e21: 0 total, 0 up, 0 in 2024-01-13T08:21:41.666 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":20,"num_osds":1,"num_up_osds":0,"osd_up_since":0,"num_in_osds":1,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:21:42.667 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:21:42.702 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: cephadm 2024-01-13T08:21:41.283790+0000 mgr.smithi099.oodult (mgr.14180) 154 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi131', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:21:42.703 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:21:42.703 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:21:42.703 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: return f(*arg) 2024-01-13T08:21:42.703 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:21:42.703 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:21:42.703 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:21:42.703 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: code, '\n'.join(err))) 2024-01-13T08:21:42.703 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:21:42.703 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp54a4qzt7:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpdi__l6lt:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:21:42.703 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:21:42.703 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:21:42.703 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:21:42.703 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new a228472f-72cf-434d-9146-7bb523644b3c 2024-01-13T08:21:42.704 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:21:42.704 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:21:42.704 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:21:42.704 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:21:42.704 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:21:42.704 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr self.prepare() 2024-01-13T08:21:42.704 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:21:42.704 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:21:42.704 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:21:42.704 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:21:42.704 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:21:42.704 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:21:42.704 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:21:42.704 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:21:42.705 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:21:42.705 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:21:42.705 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:21:42.705 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:21:42.705 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:21:42.705 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:21:42.705 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:21:42.705 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:21:42.705 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:21:42.705 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:21:42.705 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:21:42.705 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:21:42.705 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:21:42.705 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:21:42.705 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:21:42.705 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:21:42.706 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:21:42.706 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:21:42.706 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:21:42.706 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:21:42.706 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:21:42.706 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:21:42.706 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:21:42.706 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:21:42.706 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:21:42.706 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:21:42.706 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:21:42.706 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:21:42.706 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:21:42.706 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:21:42.707 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:21:42.707 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:21:42.707 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:21:42.707 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:21:42.707 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:21:42.707 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:21:42.707 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:21:42.707 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:21:42.707 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:21:42.707 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:21:42.707 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:21:42.707 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:21:42.707 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:21:42.707 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:21:42.707 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: main() 2024-01-13T08:21:42.708 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:21:42.708 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: r = ctx.func(ctx) 2024-01-13T08:21:42.708 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:21:42.708 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:21:42.708 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:21:42.708 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:21:42.708 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:21:42.708 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:21:42.708 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-8777 2024-01-13T08:21:42.709 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: 4f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:21:42.709 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:21:42.709 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:21:42.709 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:21:42.709 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:21:42.712 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:21:42.713 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp54a4qzt7:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpdi__l6lt:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:21:42.713 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: cluster 2024-01-13T08:21:41.680985+0000 mgr.smithi099.oodult (mgr.14180) 155 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:42.713 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: cephadm 2024-01-13T08:21:41.744827+0000 mgr.smithi099.oodult (mgr.14180) 156 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi131', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:21:42.713 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:21:42.713 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:21:42.713 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: return f(*arg) 2024-01-13T08:21:42.714 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:21:42.714 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:21:42.714 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:21:42.714 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: code, '\n'.join(err))) 2024-01-13T08:21:42.714 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi099/config 2024-01-13T08:21:42.715 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4wnd9y7t:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpywh7wi4g:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:21:42.715 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:21:42.715 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:21:42.715 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:21:42.715 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 25534e84-f268-442f-83bf-5bb185537503 2024-01-13T08:21:42.715 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:21:42.716 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 --yes-i-really-mean-it 2024-01-13T08:21:42.716 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr stderr: purged osd.1 2024-01-13T08:21:42.716 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:21:42.716 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:21:42.716 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr self.prepare() 2024-01-13T08:21:42.716 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:21:42.717 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:21:42.717 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:21:42.717 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:21:42.717 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:21:42.717 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:21:42.718 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:21:42.718 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:21:42.718 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:21:42.718 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:21:42.718 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:21:42.718 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:21:42.718 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:21:42.719 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:21:42.719 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:21:42.719 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:21:42.719 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:21:42.719 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:21:42.719 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:21:42.720 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:21:42.720 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:21:42.720 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:21:42.720 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:21:42.720 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:21:42.720 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:21:42.721 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:21:42.721 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:21:42.721 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:21:42.721 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:21:42.721 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:21:42.721 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:21:42.721 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:21:42.722 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:21:42.722 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:21:42.722 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:21:42.722 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:21:42.722 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:21:42.722 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:21:42.722 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr rol 2024-01-13T08:21:42.723 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: lback_osd(self.args, self.osd_id) 2024-01-13T08:21:42.723 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:21:42.723 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:21:42.723 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:21:42.723 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:21:42.723 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:21:42.723 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:21:42.724 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:21:42.724 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:21:42.724 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:21:42.724 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:21:42.724 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 1 2024-01-13T08:21:42.724 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:21:42.724 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:21:42.724 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: main() 2024-01-13T08:21:42.725 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:21:42.725 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: r = ctx.func(ctx) 2024-01-13T08:21:42.725 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:21:42.725 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:21:42.725 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:21:42.725 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:21:42.725 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:21:42.725 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:21:42.726 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:21:42.726 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:21:42.726 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:21:42.726 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:21:42.726 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:21:42.726 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:21:42.726 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4wnd9y7t:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpywh7wi4g:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:21:42.727 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: cephadm 2024-01-13T08:21:41.746049+0000 mgr.smithi099.oodult (mgr.14180) 157 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T08:21:42.727 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: service_id: all-available-devices 2024-01-13T08:21:42.727 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: service_name: osd.all-available-devices 2024-01-13T08:21:42.727 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: placement: 2024-01-13T08:21:42.727 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: host_pattern: '*' 2024-01-13T08:21:42.727 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: spec: 2024-01-13T08:21:42.728 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: data_devices: 2024-01-13T08:21:42.728 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: all: true 2024-01-13T08:21:42.728 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: filter_logic: AND 2024-01-13T08:21:42.728 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: objectstore: bluestore 2024-01-13T08:21:42.728 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:21:42.729 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp54a4qzt7:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpdi__l6lt:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:21:42.729 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:21:42.729 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:21:42.729 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:21:42.729 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new a228472f-72cf-434d-9146-7bb523644b3c 2024-01-13T08:21:42.729 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:21:42.730 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:21:42.730 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:21:42.730 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:21:42.730 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:21:42.730 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr self.prepare() 2024-01-13T08:21:42.730 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:21:42.730 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:21:42.730 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:21:42.730 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:21:42.730 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:21:42.730 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:21:42.730 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:21:42.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:21:42.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:21:42.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:21:42.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:21:42.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:21:42.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:21:42.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:21:42.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:21:42.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:21:42.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:21:42.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:21:42.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:21:42.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:21:42.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:21:42.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", 2024-01-13T08:21:42.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: line 194, in dispatch 2024-01-13T08:21:42.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:21:42.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:21:42.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:21:42.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:21:42.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:21:42.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:21:42.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:21:42.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:21:42.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:21:42.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:21:42.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:21:42.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:21:42.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:21:42.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:21:42.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:21:42.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:21:42.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:21:42.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:21:42.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:21:42.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:21:42.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:21:42.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:21:42.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:21:42.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:21:42.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:21:42.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:21:42.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:21:42.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:21:42.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:21:42.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:21:42.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: main() 2024-01-13T08:21:42.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:21:42.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: r = ctx.func(ctx) 2024-01-13T08:21:42.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:21:42.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:21:42.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:21:42.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:21:42.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:21:42.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:21:42.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:21:42.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:21:42.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:21:42.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:21:42.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:21:42.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:21:42.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp54a4qzt7:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpdi__l6lt:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:21:42.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:21:42.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T08:21:42.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: if self._apply_service(spec): 2024-01-13T08:21:42.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T08:21:42.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T08:21:42.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T08:21:42.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T08:21:42.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T08:21:42.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T08:21:42.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T08:21:42.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T08:21:42.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T08:21:42.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: raise self._value 2024-01-13T08:21:42.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T08:21:42.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: result = (True, func(*args, **kwds)) 2024-01-13T08:21:42.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T08:21:42.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: return list(map(*args)) 2024-01-13T08:21:42.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:21:42.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: return f(*arg) 2024-01-13T08:21:42.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:21:42.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:21:42.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:21:42.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: code, '\n'.join(err))) 2024-01-13T08:21:42.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:21:42.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp54a4qzt7:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpdi__l6lt:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:21:42.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:21:42.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:21:42.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:21:42.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name clie 2024-01-13T08:21:42.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: nt.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new a228472f-72cf-434d-9146-7bb523644b3c 2024-01-13T08:21:42.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:21:42.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:21:42.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:21:42.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:21:42.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:21:42.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr self.prepare() 2024-01-13T08:21:42.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:21:42.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:21:42.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:21:42.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:21:42.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:21:42.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:21:42.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:21:42.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:21:42.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:21:42.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:21:42.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:21:42.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:21:42.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:21:42.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:21:42.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:21:42.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:21:42.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:21:42.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:21:42.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:21:42.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:21:42.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:21:42.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:21:42.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:21:42.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:21:42.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:21:42.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:21:42.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:21:42.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:21:42.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:21:42.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:21:42.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:21:42.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:21:42.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:21:42.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:21:42.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:21:42.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:21:42.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:21:42.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:21:42.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:21:42.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:21:42.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:21:42.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:21:42.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:21:42.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:21:42.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:21:42.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:21:42.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:21:42.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:21:42.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:21:42.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:21:42.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:21:42.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:21:42.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: main() 2024-01-13T08:21:42.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:21:42.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: r = ctx.func(ctx) 2024-01-13T08:21:42.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:21:42.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:21:42.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:21:42.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:21:42.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:21:42.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:21:42.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:21:42.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:21:42.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:21:42.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:21:42.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:21:42.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:21:42.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp54a4qzt7:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpdi__l6lt:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:21:42.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:42 smithi099 conmon[33313]: cluster 2024-01-13T08:21:41.746665+0000 mgr.smithi099.oodult (mgr.14180) 158 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:42.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: cephadm 2024-01-13T08:21:41.283790+0000 mgr.smithi099.oodult (mgr.14180) 154 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi131', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:21:42.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:21:42.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:21:42.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: return f(*arg) 2024-01-13T08:21:42.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:21:42.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:21:42.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:21:42.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: code, '\n'.join(err))) 2024-01-13T08:21:42.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:21:42.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp54a4qzt7:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpdi__l6lt:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:21:42.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:21:42.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:21:42.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:21:42.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new a228472f-72cf-434d-9146-7bb523644b3c 2024-01-13T08:21:42.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:21:42.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:21:42.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:21:42.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:21:42.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:21:42.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr self.prepare() 2024-01-13T08:21:42.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:21:42.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:21:42.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:21:42.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:21:42.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:21:42.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:21:42.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:21:42.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:21:42.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:21:42.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:21:42.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:21:42.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:21:42.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:21:42.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:21:42.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:21:42.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:21:42.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:21:42.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:21:42.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:21:42.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:21:42.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:21:42.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:21:42.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:21:42.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:21:42.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:21:42.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:21:42.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:21:42.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:21:42.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:21:42.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:21:42.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:21:42.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:21:42.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:21:42.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:21:42.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:21:42.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:21:42.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:21:42.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:21:42.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:21:42.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:21:42.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:21:42.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:21:42.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:21:42.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:21:42.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:21:42.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:21:42.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:21:42.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:21:42.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:21:42.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:21:42.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:21:42.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:21:42.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: main() 2024-01-13T08:21:42.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:21:42.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: r = ctx.func(ctx) 2024-01-13T08:21:42.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:21:42.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:21:42.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:21:42.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:21:42.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:21:42.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:21:42.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-8777 2024-01-13T08:21:42.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: 4f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:21:42.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:21:42.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:21:42.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:21:42.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:21:42.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:21:42.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp54a4qzt7:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpdi__l6lt:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:21:42.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: cluster 2024-01-13T08:21:41.680985+0000 mgr.smithi099.oodult (mgr.14180) 155 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:42.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: cephadm 2024-01-13T08:21:41.744827+0000 mgr.smithi099.oodult (mgr.14180) 156 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi131', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:21:42.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:21:42.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:21:42.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: return f(*arg) 2024-01-13T08:21:42.907 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:21:42.907 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:21:42.907 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:21:42.907 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: code, '\n'.join(err))) 2024-01-13T08:21:42.907 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi099/config 2024-01-13T08:21:42.907 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4wnd9y7t:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpywh7wi4g:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:21:42.907 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:21:42.907 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:21:42.907 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:21:42.907 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 25534e84-f268-442f-83bf-5bb185537503 2024-01-13T08:21:42.907 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:21:42.907 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 --yes-i-really-mean-it 2024-01-13T08:21:42.907 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr stderr: purged osd.1 2024-01-13T08:21:42.907 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:21:42.907 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:21:42.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr self.prepare() 2024-01-13T08:21:42.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:21:42.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:21:42.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:21:42.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:21:42.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:21:42.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:21:42.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:21:42.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:21:42.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:21:42.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:21:42.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:21:42.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:21:42.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:21:42.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:21:42.909 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:21:42.909 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:21:42.909 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:21:42.909 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:21:42.909 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:21:42.909 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:21:42.909 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:21:42.909 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:21:42.910 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:21:42.910 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:21:42.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:21:42.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:21:42.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:21:42.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:21:42.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:21:42.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:21:42.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:21:42.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:21:42.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:21:42.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:21:42.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:21:42.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:21:42.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:21:42.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:21:42.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr rol 2024-01-13T08:21:42.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: lback_osd(self.args, self.osd_id) 2024-01-13T08:21:42.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:21:42.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:21:42.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:21:42.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:21:42.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:21:42.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:21:42.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:21:42.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:21:42.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:21:42.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:21:42.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 1 2024-01-13T08:21:42.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:21:42.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:21:42.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: main() 2024-01-13T08:21:42.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:21:42.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: r = ctx.func(ctx) 2024-01-13T08:21:42.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:21:42.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:21:42.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:21:42.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:21:42.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:21:42.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:21:42.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:21:42.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:21:42.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:21:42.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:21:42.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:21:42.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:21:42.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4wnd9y7t:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpywh7wi4g:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:21:42.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: cephadm 2024-01-13T08:21:41.746049+0000 mgr.smithi099.oodult (mgr.14180) 157 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T08:21:42.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: service_id: all-available-devices 2024-01-13T08:21:42.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: service_name: osd.all-available-devices 2024-01-13T08:21:42.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: placement: 2024-01-13T08:21:42.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: host_pattern: '*' 2024-01-13T08:21:42.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: spec: 2024-01-13T08:21:42.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: data_devices: 2024-01-13T08:21:42.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: all: true 2024-01-13T08:21:42.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: filter_logic: AND 2024-01-13T08:21:42.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: objectstore: bluestore 2024-01-13T08:21:42.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:21:42.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp54a4qzt7:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpdi__l6lt:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:21:42.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:21:42.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:21:42.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:21:42.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new a228472f-72cf-434d-9146-7bb523644b3c 2024-01-13T08:21:42.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:21:42.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:21:42.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:21:42.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:21:42.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:21:42.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr self.prepare() 2024-01-13T08:21:42.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:21:42.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:21:42.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:21:42.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:21:42.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:21:42.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:21:42.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:21:42.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:21:42.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:21:42.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:21:42.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:21:42.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:21:42.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:21:42.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:21:42.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:21:42.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:21:42.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:21:42.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:21:42.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:21:42.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:21:42.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:21:42.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", 2024-01-13T08:21:42.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: line 194, in dispatch 2024-01-13T08:21:42.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:21:42.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:21:42.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:21:42.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:21:42.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:21:42.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:21:42.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:21:42.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:21:42.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:21:42.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:21:42.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:21:42.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:21:42.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:21:42.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:21:42.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:21:42.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:21:42.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:21:42.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:21:42.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:21:42.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:21:42.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:21:42.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:21:42.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:21:42.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:21:42.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:21:42.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:21:42.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:21:42.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:21:42.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:21:42.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:21:42.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: main() 2024-01-13T08:21:42.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:21:42.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: r = ctx.func(ctx) 2024-01-13T08:21:42.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:21:42.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:21:42.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:21:42.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:21:42.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:21:42.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:21:42.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:21:42.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:21:42.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:21:42.920 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:21:42.920 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:21:42.920 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:21:42.920 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp54a4qzt7:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpdi__l6lt:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:21:42.920 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:21:42.920 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T08:21:42.920 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: if self._apply_service(spec): 2024-01-13T08:21:42.920 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T08:21:42.920 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T08:21:42.920 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T08:21:42.920 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T08:21:42.920 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T08:21:42.920 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T08:21:42.920 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T08:21:42.920 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T08:21:42.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T08:21:42.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: raise self._value 2024-01-13T08:21:42.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T08:21:42.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: result = (True, func(*args, **kwds)) 2024-01-13T08:21:42.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T08:21:42.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: return list(map(*args)) 2024-01-13T08:21:42.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:21:42.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: return f(*arg) 2024-01-13T08:21:42.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:21:42.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:21:42.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:21:42.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: code, '\n'.join(err))) 2024-01-13T08:21:42.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:21:42.921 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp54a4qzt7:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpdi__l6lt:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:21:42.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:21:42.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:21:42.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:21:42.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name clie 2024-01-13T08:21:42.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: nt.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new a228472f-72cf-434d-9146-7bb523644b3c 2024-01-13T08:21:42.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:21:42.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:21:42.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:21:42.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:21:42.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:21:42.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr self.prepare() 2024-01-13T08:21:42.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:21:42.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:21:42.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:21:42.922 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:21:42.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:21:42.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:21:42.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:21:42.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:21:42.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:21:42.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:21:42.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:21:42.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:21:42.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:21:42.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:21:42.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:21:42.923 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:21:42.924 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:21:42.924 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:21:42.924 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:21:42.925 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:21:42.925 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:21:42.925 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:21:42.925 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:21:42.925 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:21:42.925 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:21:42.925 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:21:42.925 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:21:42.925 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:21:42.925 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:21:42.925 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:21:42.925 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:21:42.925 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:21:42.925 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:21:42.925 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:21:42.925 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:21:42.926 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:21:42.926 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:21:42.926 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:21:42.926 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:21:42.926 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:21:42.926 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:21:42.926 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:21:42.926 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:21:42.926 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:21:42.926 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:21:42.926 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:21:42.926 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:21:42.926 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:21:42.926 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:21:42.926 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:21:42.927 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:21:42.927 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:21:42.927 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: main() 2024-01-13T08:21:42.927 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:21:42.927 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: r = ctx.func(ctx) 2024-01-13T08:21:42.927 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:21:42.927 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:21:42.927 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:21:42.927 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:21:42.927 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:21:42.928 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:21:42.928 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:21:42.928 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:21:42.928 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:21:42.928 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:21:42.928 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:21:42.928 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:21:42.928 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp54a4qzt7:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpdi__l6lt:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:21:42.928 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:42 smithi131 conmon[38024]: cluster 2024-01-13T08:21:41.746665+0000 mgr.smithi099.oodult (mgr.14180) 158 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:43.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:43 smithi099 conmon[33313]: cluster 2024-01-13T08:21:42.408881+0000 mon.smithi099 (mon.0 2024-01-13T08:21:43.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:43 smithi099 conmon[33313]: ) 490 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T08:21:43.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:43 smithi131 conmon[38024]: cluster 2024-01-13T08:21:42.408881+0000 mon.smithi099 (mon.0) 490 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T08:21:44.497 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:21:44.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:44 smithi131 conmon[38024]: cluster 2024-01-13T08:21:43.747137+0000 mgr.smithi099.oodult (mgr.14180) 159 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:44.676 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:44 smithi099 conmon[33313]: cluster 2024-01-13T08:21:43.747137+0000 mgr.smithi099.oodult (mgr.14180) 159 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:45.100 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:21:45.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:45 smithi099 conmon[33313]: audit 2024-01-13T08:21:44.500073+0000 mon.smithi099 (mon.0) 491 : audit [DBG] from='client.? 172.21.15.99:0/1476158483' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:21:45.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:45 smithi099 conmon[33313]: 2024-01-13T08:21:45.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:45 smithi131 conmon[38024]: audit 2024-01-13T08:21:44.500073+0000 mon.smithi099 (mon.0) 491 : audit [DBG] from='client.? 172.21.15.99:0/1476158483' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:21:46.101 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:21:46.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:46 smithi099 conmon[33313]: cluster 2024-01-13T08:21:45.747579+0000 mgr.smithi099.oodult (mgr.14180) 160 : cluster 2024-01-13T08:21:46.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:46 smithi099 conmon[33313]: [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:46.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:46 smithi131 conmon[38024]: cluster 2024-01-13T08:21:45.747579+0000 mgr.smithi099.oodult (mgr.14180) 160 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:47.928 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:21:48.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:47 smithi131 conmon[38024]: cluster 2024-01-13T08:21:47.747967+0000 mgr.smithi099.oodult (mgr.14180) 161 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:48.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:47 smithi131 conmon[38024]: 2024-01-13T08:21:48.217 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:47 smithi099 conmon[33313]: cluster 2024-01-13T08:21:47.747967+0000 mgr.smithi099.oodult (mgr.14180) 161 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:48.614 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:21:48.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:48 smithi099 conmon[33313]: audit 2024-01-13T08:21:47.931777+0000 mon.smithi099 (mon.0) 492 : audit [DBG] from='client.? 172.21.15.99:0/1020479697' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:21:48.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:48 smithi099 conmon[33313]: 2024-01-13T08:21:49.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:48 smithi131 conmon[38024]: audit 2024-01-13T08:21:47.931777+0000 mon.smithi099 (mon.0) 492 : audit [DBG] from='client.? 172.21.15.99:0/1020479697' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:21:49.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:48 smithi131 conmon[38024]: 2024-01-13T08:21:49.616 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:21:49.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:49 smithi099 conmon[33313]: cluster 2024-01-13T08:21:49.748397+0000 mgr.smithi099.oodult (mgr.14180) 162 : cluster [DBG] 2024-01-13T08:21:49.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:49 smithi099 conmon[33313]: pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:50.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:49 smithi131 conmon[38024]: cluster 2024-01-13T08:21:49.748397+0000 mgr.smithi099.oodult (mgr.14180) 162 : cluster [DBG] 2024-01-13T08:21:50.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:49 smithi131 conmon[38024]: pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:51.376 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:21:51.907 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:21:52.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:52 smithi099 conmon[33313]: audit 2024-01-13T08:21:51.380196+0000 mon.smithi099 (mon.0) 493 : audit [DBG] from='client.? 172.21.15.99:0/4102368729' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:21:52.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:52 smithi099 conmon[33313]: cluster 2024-01-13T08:21:51.748741+0000 mgr.smithi099.oodult (mgr.14180) 163 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:52.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:52 smithi131 conmon[38024]: audit 2024-01-13T08:21:51.380196+0000 mon.smithi099 (mon.0) 493 : audit [DBG] from='client.? 172.21.15.99:0/4102368729' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:21:52.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:52 smithi131 conmon[38024]: cluster 2024-01-13T08:21:51.748741+0000 mgr.smithi099.oodult (mgr.14180) 163 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:52.908 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:21:53.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:53 smithi099 conmon[33313]: cluster 2024-01-13T08:21:53.749355+0000 mgr.smithi099.oodult (mgr.14180) 164 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:54.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:53 smithi131 conmon[38024]: cluster 2024-01-13T08:21:53.749355+0000 mgr.smithi099.oodult (mgr.14180) 164 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:54.646 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:21:54.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:54 smithi099 conmon[33313]: audit 2024-01-13T08:21:54.649470+0000 mon.smithi099 (mon.0) 494 : audit [DBG] from='client.? 172.21.15.99:0/3608462203' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:21:55.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:54 smithi131 conmon[38024]: audit 2024-01-13T08:21:54.649470+0000 mon.smithi099 (mon.0) 494 : audit 2024-01-13T08:21:55.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:54 smithi131 conmon[38024]: [DBG] from='client.? 172.21.15.99:0/3608462203' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:21:55.266 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:21:56.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:55 smithi131 conmon[38024]: cluster 2024-01-13T08:21:55.749736+0000 mgr.smithi099.oodult (mgr.14180) 165 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:56.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:55 smithi131 conmon[38024]: 2024-01-13T08:21:56.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:55 smithi099 conmon[33313]: cluster 2024-01-13T08:21:55.749736+0000 mgr.smithi099.oodult (mgr.14180) 165 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:56.268 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:21:58.054 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:21:58.082 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:57 smithi099 conmon[33313]: cluster 2024-01-13T08:21:57.750141+0000 mgr.smithi099.oodult (mgr.14180) 166 : cluster [DBG] 2024-01-13T08:21:58.082 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:57 smithi099 conmon[33313]: pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:58.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:57 smithi131 conmon[38024]: cluster 2024-01-13T08:21:57.750141+0000 mgr.smithi099.oodult (mgr.14180) 166 : cluster 2024-01-13T08:21:58.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:57 smithi131 conmon[38024]: [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:21:58.606 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:21:58.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:58 smithi099 conmon[33313]: audit 2024-01-13T08:21:58.057920+0000 mon.smithi099 (mon.0) 495 : audit [DBG] from='client.? 172.21.15.99:0/1497417405' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:21:59.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:58 smithi131 conmon[38024]: audit 2024-01-13T08:21:58.057920+0000 mon.smithi099 (mon.0) 495 : audit [DBG] from='client.? 172.21.15.99:0/1497417405' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:21:59.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:58 smithi131 conmon[38024]: 2024-01-13T08:21:59.607 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:21:59.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:59 smithi099 conmon[33313]: cluster 2024-01-13T08:21:59.750500+0000 mgr.smithi099.oodult (mgr 2024-01-13T08:21:59.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:21:59 smithi099 conmon[33313]: .14180) 167 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:00.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:59 smithi131 conmon[38024]: cluster 2024-01-13T08:21:59.750500+0000 mgr.smithi099.oodult (mgr.14180) 167 : cluster [DBG] 2024-01-13T08:22:00.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:21:59 smithi131 conmon[38024]: pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:01.339 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:22:01.880 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:22:02.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:02 smithi099 conmon[33313]: audit 2024-01-13T08:22:01.342904+0000 mon.smithi099 (mon.0) 496 : audit [DBG] from='client.? 172.21.15.99:0/583751903' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:22:02.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:02 smithi099 conmon[33313]: cluster 2024-01-13T08:22:01.750883+0000 mgr.smithi099.oodult (mgr.14180) 168 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:02.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:02 smithi131 conmon[38024]: audit 2024-01-13T08:22:01.342904+0000 mon.smithi099 (mon.0) 496 : audit [DBG] from='client.? 172.21.15.99:0/583751903' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:22:02.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:02 smithi131 conmon[38024]: 2024-01-13T08:22:02.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:02 smithi131 conmon[38024]: cluster 2024-01-13T08:22:01.750883+0000 mgr.smithi099.oodult (mgr.14180) 168 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:02.882 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:22:03.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:03 smithi099 conmon[33313]: cluster 2024-01-13T08:22:03.751416+0000 mgr.smithi099.oodult (mgr.14180) 169 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:04.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:03 smithi131 conmon[38024]: cluster 2024-01-13T08:22:03.751416+0000 mgr.smithi099.oodult (mgr.14180) 169 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:04.625 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:22:04.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:04 smithi099 conmon[33313]: audit 2024-01-13T08:22:04.629677+0000 mon.smithi099 (mon.0) 497 : audit [DBG] from='client.? 172.21.15.99:0/3422014813' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:22:05.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:04 smithi131 conmon[38024]: audit 2024-01-13T08:22:04.629677+0000 mon.smithi099 (mon.0) 497 : audit [DBG] from='client.? 172.21.15.99:0/3422014813' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:22:05.162 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:22:06.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:05 smithi131 conmon[38024]: cluster 2024-01-13T08:22:05.752026+0000 mgr.smithi099.oodult (mgr.14180) 170 : cluster 2024-01-13T08:22:06.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:05 smithi131 conmon[38024]: [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:06.163 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:22:06.197 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:05 smithi099 conmon[33313]: cluster 2024-01-13T08:22:05.752026+0000 mgr.smithi099.oodult (mgr.14180) 170 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:07.952 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:22:08.121 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:07 smithi099 conmon[33313]: cluster 2024-01-13T08:22:07.752440+0000 mgr.smithi099.oodult (mgr.14180) 171 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:08.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:07 smithi131 conmon[38024]: cluster 2024-01-13T08:22:07.752440+0000 mgr.smithi099.oodult (mgr.14180) 171 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:08.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:07 smithi131 conmon[38024]: 2024-01-13T08:22:08.496 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:22:09.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:08 smithi131 conmon[38024]: audit 2024-01-13T08:22:07.956378+0000 mon.smithi099 (mon.0) 498 : audit 2024-01-13T08:22:09.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:08 smithi131 conmon[38024]: [DBG] from='client.? 172.21.15.99:0/738616205' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:22:09.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:08 smithi099 conmon[33313]: audit 2024-01-13T08:22:07.956378+0000 mon.smithi099 (mon.0) 498 : audit [DBG] from='client.? 172.21.15.99:0/738616205' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:22:09.497 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:22:10.102 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:09 smithi099 conmon[33313]: cluster 2024-01-13T08:22:09.752882+0000 mgr.smithi099.oodult (mgr.14180) 172 : cluster 2024-01-13T08:22:10.102 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:09 smithi099 conmon[33313]: [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:10.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:09 smithi131 conmon[38024]: cluster 2024-01-13T08:22:09.752882+0000 mgr.smithi099.oodult (mgr.14180) 172 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:11.280 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:22:11.844 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:22:12.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:12 smithi099 conmon[33313]: audit 2024-01-13T08:22:11.283809+0000 mon.smithi099 (mon.0 2024-01-13T08:22:12.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:12 smithi099 conmon[33313]: ) 499 : audit [DBG] from='client.? 172.21.15.99:0/1764964015' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:22:12.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:12 smithi099 conmon[33313]: cluster 2024-01-13T08:22:11.753260+0000 mgr.smithi099.oodult (mgr.14180) 173 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:12.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:12 smithi131 conmon[38024]: audit 2024-01-13T08:22:11.283809+0000 mon.smithi099 (mon.0) 499 : audit [DBG] from='client.? 172.21.15.99:0/1764964015' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:22:12.453 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:12 smithi131 conmon[38024]: cluster 2024-01-13T08:22:11.753260+0000 mgr.smithi099.oodult (mgr.14180) 173 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:12.845 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:22:13.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:13 smithi099 conmon[33313]: cluster 2024-01-13T08:22:13.753726+0000 mgr.smithi099.oodult (mgr.14180 2024-01-13T08:22:13.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:13 smithi099 conmon[33313]: ) 174 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:14.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:13 smithi131 conmon[38024]: cluster 2024-01-13T08:22:13.753726+0000 mgr.smithi099.oodult (mgr.14180) 174 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:14.675 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:22:14.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:14 smithi099 conmon[33313]: audit 2024-01-13T08:22:14.677515+0000 mon.smithi099 ( 2024-01-13T08:22:14.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:14 smithi099 conmon[33313]: mon.0) 500 : audit [DBG] from='client.? 172.21.15.99:0/1106343482' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:22:15.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:14 smithi131 conmon[38024]: audit 2024-01-13T08:22:14.677515+0000 mon.smithi099 (mon.0) 500 : audit [DBG] from='client.? 172.21.15.99:0/1106343482' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:22:15.278 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:22:16.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:15 smithi131 conmon[38024]: cluster 2024-01-13T08:22:15.754177+0000 mgr.smithi099.oodult (mgr.14180) 175 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:16.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:15 smithi099 conmon[33313]: cluster 2024-01-13T08:22:15.754177+0000 mgr.smithi099.oodult (mgr.14180) 175 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:16.279 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:22:18.038 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:22:18.066 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:17 smithi099 conmon[33313]: cluster 2024-01-13T08:22:17.754641+0000 mgr.smithi099.oodult (mgr.14180) 176 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:18.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:17 smithi131 conmon[38024]: cluster 2024-01-13T08:22:17.754641+0000 mgr.smithi099.oodult (mgr.14180) 176 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:18.626 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:22:18.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:18 smithi099 conmon[33313]: audit 2024-01-13T08:22:18.042433+0000 mon.smithi099 (mon. 2024-01-13T08:22:18.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:18 smithi099 conmon[33313]: 0) 501 : audit [DBG] from='client.? 172.21.15.99:0/2953702075' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:22:19.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:18 smithi131 conmon[38024]: audit 2024-01-13T08:22:18.042433+0000 mon.smithi099 (mon.0) 501 : audit [DBG] from='client.? 172.21.15.99:0/2953702075' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:22:19.627 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:22:19.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:19 smithi099 conmon[33313]: cluster 2024-01-13T08:22:19.755051+0000 mgr.smithi099.oodult (mgr.14180) 177 : 2024-01-13T08:22:19.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:19 smithi099 conmon[33313]: cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:20.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:19 smithi131 conmon[38024]: cluster 2024-01-13T08:22:19.755051+0000 mgr.smithi099.oodult (mgr.14180) 177 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:21.363 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:22:21.882 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:22:22.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:22 smithi099 conmon[33313]: audit 2024-01-13T08:22:21.366820+0000 mon.smithi099 (mon.0) 502 : audit [DBG] from='client.? 172.21.15.99:0/3651455989' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:22:22.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:22 smithi099 conmon[33313]: cluster 2024-01-13T08:22:21.755452+0000 mgr.smithi099.oodult (mgr.14180) 178 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:22.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:22 smithi131 conmon[38024]: audit 2024-01-13T08:22:21.366820+0000 mon.smithi099 (mon.0) 502 : audit [DBG] from='client.? 172.21.15.99:0/3651455989' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:22:22.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:22 smithi131 conmon[38024]: cluster 2024-01-13T08:22:21.755452+0000 mgr.smithi099.oodult (mgr.14180) 178 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:22.883 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:22:23.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:23 smithi099 conmon[33313]: cluster 2024-01-13T08:22:23.755852+0000 mgr.smithi099.oodult (mgr.14180) 179 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:24.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:23 smithi131 conmon[38024]: cluster 2024-01-13T08:22:23.755852+0000 mgr.smithi099.oodult (mgr.14180) 179 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:24.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:23 smithi131 conmon[38024]: 2024-01-13T08:22:24.696 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:22:24.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:24 smithi099 conmon[33313]: audit 2024-01-13T08:22:24.700415+0000 mon.smithi099 (mon. 2024-01-13T08:22:24.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:24 smithi099 conmon[33313]: 0) 503 : audit [DBG] from='client.? 172.21.15.99:0/859626880' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:22:25.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:24 smithi131 conmon[38024]: audit 2024-01-13T08:22:24.700415+0000 mon.smithi099 (mon.0) 503 : audit [DBG] from='client.? 172.21.15.99:0/859626880' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:22:25.250 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:22:26.150 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:25 smithi131 conmon[38024]: cluster 2024-01-13T08:22:25.756307+0000 mgr.smithi099.oodult (mgr.14180) 180 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:26.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:25 smithi099 conmon[33313]: cluster 2024-01-13T08:22:25.756307+0000 mgr.smithi099.oodult (mgr.14180) 180 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:26.250 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:22:28.032 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:22:28.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:27 smithi131 conmon[38024]: cluster 2024-01-13T08:22:27.756655+0000 mgr.smithi099.oodult (mgr.14180) 181 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:28.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:27 smithi099 conmon[33313]: cluster 2024-01-13T08:22:27.756655+0000 2024-01-13T08:22:28.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:27 smithi099 conmon[33313]: mgr.smithi099.oodult (mgr.14180) 181 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:28.623 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:22:28.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:28 smithi099 conmon[33313]: audit 2024-01-13T08:22:28.035714+0000 mon.smithi099 (mon. 2024-01-13T08:22:28.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:28 smithi099 conmon[33313]: 0) 504 : audit [DBG] from='client.? 172.21.15.99:0/1887644686' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:22:29.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:28 smithi131 conmon[38024]: audit 2024-01-13T08:22:28.035714+0000 mon.smithi099 (mon.0) 504 : audit [DBG] from='client.? 172.21.15.99:0/1887644686' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:22:29.624 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:22:29.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:29 smithi099 conmon[33313]: cluster 2024-01-13T08:22:29.757050+0000 mgr.smithi099.oodult (mgr.14180) 182 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:30.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:29 smithi131 conmon[38024]: cluster 2024-01-13T08:22:29.757050+0000 mgr.smithi099.oodult (mgr.14180) 182 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:31.444 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:22:32.034 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:22:32.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:32 smithi131 conmon[38024]: audit 2024-01-13T08:22:31.448712+0000 mon.smithi099 (mon.0) 505 : audit [DBG] from='client.? 172.21.15.99:0/1767636650' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:22:32.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:32 smithi131 conmon[38024]: cluster 2024-01-13T08:22:31.757760+0000 mgr.smithi099.oodult (mgr.14180) 183 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:32.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:32 smithi099 conmon[33313]: audit 2024-01-13T08:22:31.448712+0000 mon.smithi099 (mon 2024-01-13T08:22:32.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:32 smithi099 conmon[33313]: .0) 505 : audit [DBG] from='client.? 172.21.15.99:0/1767636650' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:22:32.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:32 smithi099 conmon[33313]: cluster 2024-01-13T08:22:31.757760+0000 mgr.smithi099.oodult (mgr.14180) 183 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:33.035 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:22:34.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:33 smithi131 conmon[38024]: cluster 2024-01-13T08:22:33.758219+0000 mgr.smithi099.oodult (mgr.14180) 184 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:34.198 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:33 smithi099 conmon[33313]: cluster 2024-01-13T08:22:33.758219+0000 mgr.smithi099.oodult (mgr.14180) 184 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:34.782 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:22:35.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:34 smithi131 conmon[38024]: audit 2024-01-13T08:22:34.786747+0000 mon.smithi099 (mon.0) 506 : audit [DBG] from='client.? 172.21.15.99:0/3209824213' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:22:35.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:34 smithi131 conmon[38024]: 2024-01-13T08:22:35.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:34 smithi099 conmon[33313]: audit 2024-01-13T08:22:34.786747+0000 mon.smithi099 (mon.0) 506 : audit [DBG] from='client.? 172.21.15.99:0/3209824213' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:22:35.338 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:22:36.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:35 smithi131 conmon[38024]: cluster 2024-01-13T08:22:35.758683+0000 mgr.smithi099.oodult (mgr.14180) 185 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:36.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:35 smithi131 conmon[38024]: 2024-01-13T08:22:36.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:35 smithi099 conmon[33313]: cluster 2024-01-13T08:22:35.758683+0000 mgr.smithi099.oodult (mgr.14180) 185 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:36.339 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:22:38.122 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:22:38.149 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:37 smithi099 conmon[33313]: cluster 2024-01-13T08:22:37.759049+0000 mgr.smithi099.oodult (mgr.14180) 186 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:38.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:37 smithi131 conmon[38024]: cluster 2024-01-13T08:22:37.759049+0000 mgr.smithi099.oodult (mgr.14180) 186 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:38.890 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:22:39.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:38 smithi131 conmon[38024]: audit 2024-01-13T08:22:38.126476+0000 mon.smithi099 (mon.0) 507 : audit [DBG] from='client.? 172.21.15.99:0/1677711707' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:22:39.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:38 smithi099 conmon[33313]: audit 2024-01-13T08:22:38.126476+0000 mon.smithi099 (mon.0) 507 : audit [DBG] from='client.? 172.21.15.99:0/1677711707' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:22:39.891 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:22:40.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:39 smithi131 conmon[38024]: cluster 2024-01-13T08:22:39.759654+0000 mgr.smithi099.oodult (mgr.14180) 187 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:40.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:39 smithi099 conmon[33313]: cluster 2024-01-13T08:22:39.759654+0000 mgr.smithi099.oodult (mgr.14180) 187 : cluster [DBG] 2024-01-13T08:22:40.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:39 smithi099 conmon[33313]: pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:41.620 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:22:42.171 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:22:42.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:42 smithi131 conmon[38024]: audit 2024-01-13T08:22:41. 2024-01-13T08:22:42.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:42 smithi131 conmon[38024]: 624136+0000 mon.smithi099 (mon.0) 508 : audit [DBG] from='client.? 172.21.15.99:0/237570080' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:22:42.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:42 smithi131 conmon[38024]: audit 2024-01-13T08:22:41.747806+0000 mon.smithi099 (mon.0) 509 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:22:42.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:42 smithi131 conmon[38024]: cluster 2024-01-13T08:22:41.760208+0000 mgr.smithi099.oodult (mgr.14180) 188 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:42.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:42 smithi099 conmon[33313]: audit 2024-01-13T08:22:41.624136+0000 mon.smithi099 (mon.0) 508 : audit [DBG] from='client.? 172.21.15.99:0/237570080' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:22:42.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:42 smithi099 conmon[33313]: audit 2024-01-13T08:22:41.747806+0000 mon.smithi099 (mon.0) 509 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:22:42.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:42 smithi099 conmon[33313]: cluster 2024-01-13T08:22:41.760208+0000 mgr.smithi099.oodult (mgr.14180) 188 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:43.172 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:22:43.292 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:43 smithi131 conmon[38024]: audit 2024-01-13T08:22:42.087332+0000 mon.smithi099 (mon.0) 510 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:22:43.292 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:43 smithi131 conmon[38024]: audit 2024-01-13T08:22:42.087932+0000 mon.smithi099 (mon.0) 511 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T08:22:43.292 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:43 smithi131 conmon[38024]: audit 2024-01-13T08:22:42.091817+0000 mon.smithi099 (mon.0) 512 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:22:43.292 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:43 smithi131 conmon[38024]: cluster 2024-01-13T08:22:42.092125+0000 mgr.smithi099.oodult (mgr.14180) 189 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:43.292 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:43 smithi131 conmon[38024]: audit 2024-01-13T08:22:42.095447+0000 mon.smithi099 (mon.0) 513 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T08:22:43.292 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:43 smithi131 conmon[38024]: audit 2024-01-13T08:22:42.098434+0000 mon.smithi099 (mon.0) 514 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:22:43.292 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:43 smithi131 conmon[38024]: audit 2024-01-13T08:22:42.099448+0000 mon.smithi099 (mon.0) 515 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:22:43.292 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:43 smithi131 conmon[38024]: audit 2024-01-13T08:22:42.099606+0000 mon.smithi099 (mon.0) 516 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:22:43.292 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:43 smithi131 conmon[38024]: audit 2024-01-13T08:22:42.100001+0000 mon.smithi099 (mon.0) 517 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:22:43.292 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:43 smithi099 conmon[33313]: audit 2024-01-13T08:22:42.087332+0000 mon.smithi099 (mon.0) 510 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:22:43.292 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:43 smithi099 conmon[33313]: audit 2024-01-13T08:22:42.087932+0000 mon.smithi099 (mon.0) 511 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T08:22:43.293 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:43 smithi099 conmon[33313]: audit 2024-01-13T08:22:42.091817+0000 mon.smithi099 (mon.0) 512 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:22:43.293 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:43 smithi099 conmon[33313]: cluster 2024-01-13T08:22:42.092125+0000 mgr.smithi099.oodult (mgr.14180) 189 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:43.293 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:43 smithi099 conmon[33313]: audit 2024-01-13T08:22:42.095447+0000 mon.smithi099 (mon.0) 513 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T08:22:43.293 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:43 smithi099 conmon[33313]: audit 2024-01-13T08:22:42.098434+0000 mon.smithi099 (mon.0) 514 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:22:43.293 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:43 smithi099 conmon[33313]: audit 2024-01-13T08:22:42.099448+0000 mon.smithi099 (mon.0) 515 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:22:43.293 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:43 smithi099 conmon[33313]: audit 2024-01-13T08:22:42.099606+0000 mon.smithi099 (mon.0) 516 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:22:43.293 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:43 smithi099 conmon[33313]: 2024-01-13T08:22:43.293 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:43 smithi099 conmon[33313]: audit 2024-01-13T08:22:42.100001+0000 mon.smithi099 (mon.0) 517 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:22:44.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:44 smithi099 conmon[33313]: cluster 2024-01-13T08:22:43.090211+0000 mon.smithi099 (mon.0) 518 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T08:22:44.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:44 smithi099 conmon[33313]: cluster 2024-01-13T08:22:43.090232+0000 mon.smithi099 (mon.0) 519 : cluster [INF] Cluster is now healthy 2024-01-13T08:22:44.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:44 smithi131 conmon[38024]: cluster 2024-01-13T08:22:43.090211+0000 mon.smithi099 (mon.0) 518 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T08:22:44.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:44 smithi131 conmon[38024]: cluster 2024-01-13T08:22:43.090232+0000 mon.smithi099 (mon.0) 519 : cluster [INF] Cluster is now healthy 2024-01-13T08:22:45.060 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:45 smithi099 conmon[33313]: cluster 2024-01-13T08:22:44.092358+0000 mgr.smithi099.oodult (mgr.14180) 190 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:45.060 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:45 smithi099 conmon[33313]: audit 2024-01-13T08:22:44.611981+0000 mon.smithi131 (mon.1) 10 : audit [INF] from='client.? 172.21.15.131:0/678732110' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e81c801a-c853-48bd-9922-8a3c006751d8"}]: dispatch 2024-01-13T08:22:45.061 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:45 smithi099 conmon[33313]: audit 2024-01-13T08:22:44.612393+0000 mon.smithi099 (mon.0) 520 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e81c801a-c853-48bd-9922-8a3c006751d8"}]: dispatch 2024-01-13T08:22:45.061 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:45 smithi099 conmon[33313]: audit 2024-01-13T08:22:44.617050+0000 mon.smithi099 (mon.0) 521 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e81c801a-c853-48bd-9922-8a3c006751d8"}]': finished 2024-01-13T08:22:45.061 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:45 smithi099 conmon[33313]: cluster 2024-01-13T08:22:44.617121+0000 mon.smithi099 (mon.0) 522 : cluster [DBG] osdmap e22: 1 total, 0 up, 1 in 2024-01-13T08:22:45.061 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:45 smithi099 conmon[33313]: audit 2024-01-13T08:22:44.617241+0000 mon.smithi099 (mon.0) 523 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:22:45.061 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:45 smithi099 conmon[33313]: audit 2024-01-13T08:22:44.702822+0000 mon.smithi099 (mon.0) 524 : audit [INF] from='client.? 172.21.15.99:0/3523667290' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "151137f1-33eb-41ee-add5-d2b418ff3244"}]: dispatch 2024-01-13T08:22:45.061 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:45 smithi099 conmon[33313]: audit 2024-01-13T08:22:44.706190+0000 mon.smithi099 (mon.0) 525 : audit [INF] from='client.? 172.21.15.99:0/3523667290' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "151137f1-33eb-41ee-add5-d2b418ff3244"}]': finished 2024-01-13T08:22:45.061 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:45 smithi099 conmon[33313]: cluster 2024-01-13T08:22:44.706230+0000 mon.smithi099 (mon.0) 526 : cluster [DBG] osdmap e23: 2 total, 0 up, 2 in 2024-01-13T08:22:45.061 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:45 smithi099 conmon[33313]: audit 2024-01-13T08:22:44.706341+0000 mon.smithi099 (mon.0) 527 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:22:45.061 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:45 smithi099 conmon[33313]: audit 2024-01-13T08:22:44.706640+0000 mon.smithi099 (mon.0) 528 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-13T08:22:45.061 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:45 smithi099 conmon[33313]: audit 2024-01-13T08:22:44.903542+0000 mon.smithi131 (mon.1) 11 : audit [INF] from='client.? 172.21.15.131:0/1039957193' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:22:45.061 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:45 smithi099 conmon[33313]: audit 2024-01-13T08:22:44.903951+0000 mon.smithi099 (mon.0) 529 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:22:45.061 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:45 smithi099 conmon[33313]: audit 2024-01-13T08:22:44.907763+0000 mon.smithi099 (mon.0) 530 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:22:45.061 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:45 smithi099 conmon[33313]: cluster 2024-01-13T08:22:44.907809+0000 mon.smithi099 (mon.0) 531 : cluster [DBG] osdmap e24: 1 total, 0 up, 1 in 2024-01-13T08:22:45.062 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:45 smithi099 conmon[33313]: audit 2024-01-13T08:22:44.907925+0000 mon.smithi099 (mon.0) 532 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-13T08:22:45.062 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:45 smithi099 conmon[33313]: audit 2024-01-13T08:22:45.062 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:45 smithi099 conmon[33313]: 2024-01-13T08:22:45.016962+0000 mon.smithi099 (mon.0) 533 : audit [INF] from='client.? 172.21.15.99:0/1999120314' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:22:45.062 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:45 smithi099 conmon[33313]: audit 2024-01-13T08:22:45.020602+0000 mon.smithi099 (mon.0) 534 : audit [INF] from='client.? 172.21.15.99:0/1999120314' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:22:45.062 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:45 smithi099 conmon[33313]: cluster 2024-01-13T08:22:45.062 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:45 smithi099 conmon[33313]: 2024-01-13T08:22:45.020663+0000 mon.smithi099 ( 2024-01-13T08:22:45.062 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:45 smithi099 conmon[33313]: mon.0) 535 : cluster [DBG] osdmap e25: 0 total, 0 up, 0 in 2024-01-13T08:22:45.062 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:45 smithi099 conmon[33313]: 2024-01-13T08:22:45.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: cluster 2024-01-13T08:22:44.092358+0000 mgr.smithi099.oodult (mgr.14180) 190 2024-01-13T08:22:45.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:45.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: audit 2024-01-13T08:22:44.611981+0000 mon.smithi131 (mon.1) 10 : audit [INF] from='client.? 172.21.15.131:0/678732110' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e81c801a-c853-48bd-9922-8a3c006751d8"}]: dispatch 2024-01-13T08:22:45.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: audit 2024-01-13T08:22:44.612393+0000 mon.smithi099 (mon.0) 520 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e81c801a-c853-48bd-9922-8a3c006751d8"}]: dispatch 2024-01-13T08:22:45.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: audit 2024-01-13T08:22:44.617050+0000 mon.smithi099 (mon.0) 521 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e81c801a-c853-48bd-9922-8a3c006751d8"}]': finished 2024-01-13T08:22:45.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: cluster 2024-01-13T08:22:44.617121+0000 mon.smithi099 (mon.0) 522 : cluster [DBG] osdmap e22: 1 total, 0 up, 1 in 2024-01-13T08:22:45.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: audit 2024-01-13T08:22:44.617241+0000 mon.smithi099 (mon.0) 523 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:22:45.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: audit 2024-01-13T08: 2024-01-13T08:22:45.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: 22:44.702822+0000 mon.smithi099 (mon.0) 524 2024-01-13T08:22:45.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: : audit [INF] from='client.? 172.21.15.99:0/3523667290' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "151137f1-33eb-41ee-add5-d2b418ff3244"}]: dispatch 2024-01-13T08:22:45.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: audit 2024-01-13T08:22 2024-01-13T08:22:45.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: :44.706190+0000 mon.smithi099 (mon.0) 525 : 2024-01-13T08:22:45.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: audit [INF] from='client.? 172.21.15.99:0/3523667290' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "151137f1-33eb-41ee-add5-d2b418ff3244"}]': finished 2024-01-13T08:22:45.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: cluster 2024-01-13T08:22 2024-01-13T08:22:45.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: :44.706230+0000 mon.smithi099 (mon.0) 526 : 2024-01-13T08:22:45.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: cluster [DBG] osdmap e23: 2 total, 0 up, 2 in 2024-01-13T08:22:45.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: audit 2024-01-13T08:22: 2024-01-13T08:22:45.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: 44.706341+0000 mon.smithi099 (mon.0) 527 : audit 2024-01-13T08:22:45.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:22:45.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: audit 2024-01-13T08:22 2024-01-13T08:22:45.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: :44.706640+0000 mon.smithi099 (mon.0) 528 : 2024-01-13T08:22:45.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-13T08:22:45.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: audit 2024-01-13T08:22: 2024-01-13T08:22:45.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: 44.903542+0000 mon.smithi131 (mon.1) 11 : 2024-01-13T08:22:45.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: audit [INF] from='client.? 172.21.15.131:0/1039957193' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:22:45.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: audit 2024-01-13T08:22 2024-01-13T08:22:45.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: :44.903951+0000 mon.smithi099 (mon.0) 529 2024-01-13T08:22:45.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:22:45.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: audit 2024-01-13T08:22:44 2024-01-13T08:22:45.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: .907763+0000 mon.smithi099 (mon.0) 530 : audit [INF] 2024-01-13T08:22:45.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:22:45.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: cluster 2024-01-13T08:22:44.907809 2024-01-13T08:22:45.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: +0000 mon.smithi099 (mon. 2024-01-13T08:22:45.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: 0) 531 : cluster [DBG] osdmap e24: 1 total, 0 up, 1 in 2024-01-13T08:22:45.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: 2024-01-13T08:22:45.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: audit 2024 2024-01-13T08:22:45.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: -01-13T08:22:44.907925+0000 mon.smithi099 ( 2024-01-13T08:22:45.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: mon.0) 532 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-13T08:22:45.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: audit 2024- 2024-01-13T08:22:45.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: 01-13T08:22:45.016962+0000 mon.smithi099 2024-01-13T08:22:45.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: (mon.0) 533 : audit [INF] from='client.? 172.21.15.99:0/1999120314' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:22:45.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: audit 2024- 2024-01-13T08:22:45.403 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: 01-13T08:22:45.020602+0000 mon.smithi099 ( 2024-01-13T08:22:45.403 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: mon.0) 534 2024-01-13T08:22:45.403 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: : audit [INF] from='client.? 172.21.15.99:0/1999120314' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:22:45.403 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: cluster 2024-01-13T08: 2024-01-13T08:22:45.403 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: 22:45.020663+0000 mon.smithi099 (mon.0) 535 2024-01-13T08:22:45.403 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:45 smithi131 conmon[38024]: : cluster [DBG] osdmap e25: 0 total, 0 up, 0 in 2024-01-13T08:22:45.410 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:22:46.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: audit 2024-01-13T08:22:45.413802+0000 mon.smithi099 (mon.0) 536 : audit [DBG] from='client.? 172.21.15.99:0/3384018096' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:22:46.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: cephadm 2024-01-13T08:22:45.631051+0000 mgr.smithi099.oodult ( 2024-01-13T08:22:46.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: mgr.14180) 191 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi131', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:22:46.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:22:46.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:22:46.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: return f(*arg) 2024-01-13T08:22:46.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:22:46.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:22:46.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:22:46.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: code, '\n'.join(err))) 2024-01-13T08:22:46.232 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:22:46.233 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpstfo06z6:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbriqp69y:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:22:46.233 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:22:46.233 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:22:46.233 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:22:46.233 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new e81c801a-c853-48bd-9922-8a3c006751d8 2024-01-13T08:22:46.233 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:22:46.233 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:22:46.233 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:22:46.233 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:22:46.233 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:22:46.233 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr self.prepare() 2024-01-13T08:22:46.233 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:22:46.233 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:22:46.233 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:22:46.233 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:22:46.234 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:22:46.234 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:22:46.234 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:22:46.234 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:22:46.234 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:22:46.234 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:22:46.234 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:22:46.234 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:22:46.234 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:22:46.234 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:22:46.234 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:22:46.234 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:22:46.234 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:22:46.234 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:22:46.234 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:22:46.235 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:22:46.235 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:22:46.235 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:22:46.235 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:22:46.235 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:22:46.235 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:22:46.235 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:22:46.235 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:22:46.235 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:22:46.235 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:22:46.235 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:22:46.235 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:22:46.235 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:22:46.235 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:22:46.236 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:22:46.236 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:22:46.236 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:22:46.236 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:22:46.236 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:22:46.236 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:22:46.236 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:22:46.236 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:22:46.236 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:22:46.236 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:22:46.236 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:22:46.236 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:22:46.236 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:22:46.236 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:22:46.237 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:22:46.237 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:22:46.237 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:22:46.237 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:22:46.237 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:22:46.237 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: main() 2024-01-13T08:22:46.237 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:22:46.237 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: r = ctx.func(ctx) 2024-01-13T08:22:46.237 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:22:46.237 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:22:46.237 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:22:46.237 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:22:46.237 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:22:46.237 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:22:46.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39fe 2024-01-13T08:22:46.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: bbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:22:46.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:22:46.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:22:46.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:22:46.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:22:46.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:22:46.238 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:46 smithi099 conmon[33313]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpstfo06z6:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbriqp69y:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:22:46.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: audit 2024-01-13T08:22:45.413802+0000 mon.smithi099 (mon.0) 536 : audit [DBG] from='client.? 172.21.15.99:0/3384018096' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:22:46.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: cephadm 2024-01-13T08:22:45.631051+0000 mgr.smithi099.oodult (mgr.14180) 191 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi131', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:22:46.400 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:22:46.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:22:46.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: return f(*arg) 2024-01-13T08:22:46.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:22:46.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:22:46.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:22:46.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: code, '\n'.join(err))) 2024-01-13T08:22:46.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:22:46.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpstfo06z6:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbriqp69y:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:22:46.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:22:46.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:22:46.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:22:46.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new e81c801a-c853-48bd-9922-8a3c006751d8 2024-01-13T08:22:46.403 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:22:46.403 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:22:46.403 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:22:46.403 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:22:46.403 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:22:46.404 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr self.prepare() 2024-01-13T08:22:46.404 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:22:46.404 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:22:46.404 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:22:46.404 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:22:46.404 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:22:46.405 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:22:46.405 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:22:46.405 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:22:46.405 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:22:46.405 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:22:46.405 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:22:46.406 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:22:46.406 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:22:46.406 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:22:46.406 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:22:46.406 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:22:46.406 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:22:46.407 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:22:46.407 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:22:46.407 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:22:46.407 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:22:46.407 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:22:46.408 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:22:46.408 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:22:46.408 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:22:46.408 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:22:46.408 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:22:46.408 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:22:46.409 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:22:46.409 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:22:46.409 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:22:46.409 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:22:46.409 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:22:46.409 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:22:46.410 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:22:46.410 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:22:46.410 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:22:46.410 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:22:46.410 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:22:46.410 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:22:46.411 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:22:46.411 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:22:46.411 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:22:46.411 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:22:46.411 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:22:46.411 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:22:46.412 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:22:46.412 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:22:46.412 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:22:46.412 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:22:46.412 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:22:46.412 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:22:46.413 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: main() 2024-01-13T08:22:46.413 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:22:46.413 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: r = ctx.func(ctx) 2024-01-13T08:22:46.413 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:22:46.413 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:22:46.414 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:22:46.414 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:22:46.414 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1 2024-01-13T08:22:46.414 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:22:46.415 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:22:46.415 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:22:46.415 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:22:46.415 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:22:46.415 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:22:46.415 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:22:46.416 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:22:46.416 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:46 smithi131 conmon[38024]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpstfo06z6:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbriqp69y:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:22:46.563 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:22:47.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: cephadm 2024-01-13T08:22:45.988819+0000 mgr.smithi099.oodult (mgr.14180) 192 : 2024-01-13T08:22:47.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi131', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:22:47.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:22:47.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:22:47.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: return f(*arg) 2024-01-13T08:22:47.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:22:47.401 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:22:47.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:22:47.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: code, '\n'.join(err))) 2024-01-13T08:22:47.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi099/config 2024-01-13T08:22:47.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpdtgbakso:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpcu1ygzqf:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:22:47.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:22:47.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:22:47.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:22:47.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 151137f1-33eb-41ee-add5-d2b418ff3244 2024-01-13T08:22:47.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:22:47.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 --yes-i-really-mean-it 2024-01-13T08:22:47.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr stderr: purged osd.1 2024-01-13T08:22:47.402 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:22:47.403 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:22:47.403 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr self.prepare() 2024-01-13T08:22:47.403 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:22:47.403 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:22:47.403 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:22:47.403 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:22:47.404 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:22:47.404 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:22:47.404 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:22:47.404 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:22:47.404 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:22:47.404 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:22:47.404 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:22:47.404 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:22:47.404 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:22:47.404 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:22:47.405 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:22:47.405 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:22:47.405 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:22:47.405 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:22:47.405 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:22:47.405 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:22:47.405 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:22:47.405 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:22:47.405 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:22:47.405 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:22:47.405 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:22:47.405 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:22:47.405 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:22:47.405 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:22:47.405 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:22:47.406 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:22:47.406 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:22:47.406 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:22:47.406 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:22:47.406 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:22:47.406 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:22:47.406 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:22:47.406 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:22:47.406 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:22:47.406 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:22:47.406 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:22:47.406 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:22:47.406 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:22:47.406 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:22:47.406 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:22:47.407 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:22:47.407 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:22:47.407 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:22:47.407 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:22:47.407 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:22:47.407 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 1 2024-01-13T08:22:47.407 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:22:47.407 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:22:47.407 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: main() 2024-01-13T08:22:47.407 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:22:47.407 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: r = ctx.func(ctx) 2024-01-13T08:22:47.407 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:22:47.407 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:22:47.407 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:22:47.407 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:22:47.408 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:22:47.408 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:22:47.408 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90 2024-01-13T08:22:47.408 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: d2", line 1927, in _validate_fsid 2024-01-13T08:22:47.408 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:22:47.408 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:22:47.408 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:22:47.408 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:22:47.408 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:22:47.408 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpdtgbakso:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpcu1ygzqf:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:22:47.408 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: cephadm 2024-01-13T08:22:45.989980+0000 mgr.smithi099.oodult (mgr.14180) 193 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T08:22:47.408 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: service_id: all-available-devices 2024-01-13T08:22:47.408 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: service_name: osd.all-available-devices 2024-01-13T08:22:47.409 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: placement: 2024-01-13T08:22:47.409 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: host_pattern: '*' 2024-01-13T08:22:47.409 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: spec: 2024-01-13T08:22:47.409 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: data_devices: 2024-01-13T08:22:47.409 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: all: true 2024-01-13T08:22:47.409 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: filter_logic: AND 2024-01-13T08:22:47.409 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: objectstore: bluestore 2024-01-13T08:22:47.409 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:22:47.409 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpstfo06z6:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbriqp69y:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:22:47.409 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:22:47.409 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:22:47.409 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:22:47.409 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new e81c801a-c853-48bd-9922-8a3c006751d8 2024-01-13T08:22:47.409 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:22:47.410 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:22:47.410 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:22:47.410 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:22:47.410 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:22:47.410 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr self.prepare() 2024-01-13T08:22:47.410 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:22:47.410 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:22:47.410 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:22:47.410 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:22:47.410 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:22:47.410 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:22:47.410 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:22:47.410 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:22:47.410 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:22:47.410 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:22:47.410 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:22:47.411 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:22:47.411 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:22:47.411 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:22:47.411 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:22:47.411 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:22:47.411 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:22:47.411 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:22:47.411 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:22:47.411 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:22:47.411 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:22:47.411 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:22:47.412 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:22:47.412 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:22:47.412 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:22:47.412 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:22:47.412 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:22:47.412 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:22:47.412 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:22:47.412 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:22:47.412 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:22:47.412 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:22:47.412 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:22:47.412 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:22:47.412 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:22:47.412 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:22:47.412 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:22:47.413 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:22:47.413 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:22:47.413 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:22:47.413 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:22:47.413 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:22:47.413 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:22:47.413 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:22:47.413 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:22:47.413 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:22:47.413 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr devices = find_associ 2024-01-13T08:22:47.413 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: ated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:22:47.413 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:22:47.413 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:22:47.413 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:22:47.413 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:22:47.414 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:22:47.414 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: main() 2024-01-13T08:22:47.414 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:22:47.414 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: r = ctx.func(ctx) 2024-01-13T08:22:47.414 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:22:47.414 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:22:47.414 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:22:47.415 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:22:47.415 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:22:47.415 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:22:47.415 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:22:47.415 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:22:47.415 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:22:47.415 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:22:47.415 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:22:47.415 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:22:47.415 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpstfo06z6:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbriqp69y:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:22:47.415 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:22:47.415 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T08:22:47.415 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: if self._apply_service(spec): 2024-01-13T08:22:47.415 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T08:22:47.416 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T08:22:47.416 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T08:22:47.416 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T08:22:47.416 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T08:22:47.416 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T08:22:47.416 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T08:22:47.416 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T08:22:47.416 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T08:22:47.416 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: raise self._value 2024-01-13T08:22:47.416 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T08:22:47.416 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: result = (True, func(*args, **kwds)) 2024-01-13T08:22:47.416 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T08:22:47.416 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: return list(map(*args)) 2024-01-13T08:22:47.416 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:22:47.416 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: return f(*arg) 2024-01-13T08:22:47.417 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:22:47.417 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:22:47.417 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:22:47.417 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: code, '\n'.join(err))) 2024-01-13T08:22:47.417 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:22:47.417 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpstfo06z6:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbriqp69y:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:22:47.417 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:22:47.417 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:22:47.417 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:22:47.417 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new e81c801a-c853-48bd-9922-8a3c006751d8 2024-01-13T08:22:47.417 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:22:47.417 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:22:47.417 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:22:47.417 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:22:47.418 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:22:47.418 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr self.prepare() 2024-01-13T08:22:47.418 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:22:47.418 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:22:47.418 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:22:47.418 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:22:47.418 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:22:47.418 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:22:47.418 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:22:47.418 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:22:47.418 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:22:47.418 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:22:47.418 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:22:47.418 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:22:47.418 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:22:47.419 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:22:47.419 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:22:47.419 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:22:47.419 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:22:47.419 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in 2024-01-13T08:22:47.419 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: newfunc 2024-01-13T08:22:47.419 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:22:47.419 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:22:47.419 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:22:47.419 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:22:47.419 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:22:47.419 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:22:47.420 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:22:47.420 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:22:47.420 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:22:47.420 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:22:47.420 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:22:47.420 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:22:47.420 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:22:47.420 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:22:47.420 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:22:47.420 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:22:47.420 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:22:47.420 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:22:47.421 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:22:47.421 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:22:47.421 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:22:47.421 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:22:47.421 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:22:47.421 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:22:47.421 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:22:47.422 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:22:47.422 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:22:47.422 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:22:47.422 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:22:47.422 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:22:47.422 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:22:47.422 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:22:47.424 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:22:47.424 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:22:47.424 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: main() 2024-01-13T08:22:47.424 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:22:47.424 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: r = ctx.func(ctx) 2024-01-13T08:22:47.424 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:22:47.424 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:22:47.424 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:22:47.424 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:22:47.424 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:22:47.425 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:22:47.425 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:22:47.425 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:22:47.425 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:22:47.425 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:22:47.425 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:22:47.425 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:22:47.425 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpstfo06z6:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbriqp69y:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:22:47.425 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: cluster 2024-01-13T08:22:45.990683+0000 mgr.smithi099.oodult (mgr.14180) 194 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:47.425 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:47 smithi131 conmon[38024]: cluster 2024-01-13T08:22:46.032649+0000 mon.smithi099 (mon.0) 537 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T08:22:47.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: cephadm 2024-01-13T08:22:45.988819+0000 mgr.smithi099.oodult (mgr.14180) 192 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi131', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:22:47.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:22:47.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:22:47.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: return f(*arg) 2024-01-13T08:22:47.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:22:47.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:22:47.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:22:47.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: code, '\n'.join(err))) 2024-01-13T08:22:47.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi099/config 2024-01-13T08:22:47.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpdtgbakso:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpcu1ygzqf:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:22:47.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:22:47.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:22:47.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:22:47.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 151137f1-33eb-41ee-add5-d2b418ff3244 2024-01-13T08:22:47.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:22:47.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 --yes-i-really-mean-it 2024-01-13T08:22:47.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr stderr: purged osd.1 2024-01-13T08:22:47.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:22:47.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:22:47.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr self.prepare() 2024-01-13T08:22:47.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:22:47.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:22:47.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:22:47.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:22:47.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:22:47.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:22:47.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:22:47.483 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:22:47.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:22:47.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:22:47.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:22:47.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:22:47.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:22:47.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:22:47.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:22:47.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:22:47.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:22:47.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:22:47.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:22:47.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:22:47.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:22:47.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:22:47.484 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:22:47.485 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:22:47.485 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:22:47.485 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:22:47.485 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:22:47.485 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:22:47.485 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:22:47.485 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:22:47.485 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:22:47.485 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:22:47.485 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:22:47.485 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:22:47.485 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:22:47.485 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:22:47.486 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:22:47.486 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:22:47.486 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:22:47.486 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:22:47.486 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:22:47.486 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:22:47.486 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:22:47.486 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:22:47.486 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:22:47.486 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:22:47.486 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:22:47.486 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:22:47.486 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:22:47.486 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 1 2024-01-13T08:22:47.486 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:22:47.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:22:47.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: main() 2024-01-13T08:22:47.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:22:47.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: r = ctx.func(ctx) 2024-01-13T08:22:47.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:22:47.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:22:47.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:22:47.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:22:47.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:22:47.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:22:47.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-8777 2024-01-13T08:22:47.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: 4f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:22:47.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:22:47.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:22:47.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:22:47.487 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:22:47.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:22:47.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpdtgbakso:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpcu1ygzqf:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:22:47.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: cephadm 2024-01-13T08:22:45.989980+0000 mgr.smithi099.oodult (mgr.14180) 193 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T08:22:47.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: service_id: all-available-devices 2024-01-13T08:22:47.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: service_name: osd.all-available-devices 2024-01-13T08:22:47.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: placement: 2024-01-13T08:22:47.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: host_pattern: '*' 2024-01-13T08:22:47.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: spec: 2024-01-13T08:22:47.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: data_devices: 2024-01-13T08:22:47.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: all: true 2024-01-13T08:22:47.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: filter_logic: AND 2024-01-13T08:22:47.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: objectstore: bluestore 2024-01-13T08:22:47.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:22:47.488 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpstfo06z6:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbriqp69y:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:22:47.489 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:22:47.489 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:22:47.489 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:22:47.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new e81c801a-c853-48bd-9922-8a3c006751d8 2024-01-13T08:22:47.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:22:47.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:22:47.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:22:47.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:22:47.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:22:47.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr self.prepare() 2024-01-13T08:22:47.490 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:22:47.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:22:47.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:22:47.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:22:47.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:22:47.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:22:47.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:22:47.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:22:47.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:22:47.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:22:47.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:22:47.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:22:47.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:22:47.491 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:22:47.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:22:47.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:22:47.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:22:47.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:22:47.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:22:47.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:22:47.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:22:47.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:22:47.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:22:47.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:22:47.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:22:47.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:22:47.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:22:47.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:22:47.492 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:22:47.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:22:47.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:22:47.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:22:47.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:22:47.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:22:47.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:22:47.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:22:47.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:22:47.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:22:47.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:22:47.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:22:47.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:22:47.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:22:47.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:22:47.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:22:47.493 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:22:47.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/ 2024-01-13T08:22:47.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: lvm/zap.py", line 302, in zap_osd 2024-01-13T08:22:47.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:22:47.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:22:47.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:22:47.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:22:47.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:22:47.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:22:47.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: main() 2024-01-13T08:22:47.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:22:47.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: r = ctx.func(ctx) 2024-01-13T08:22:47.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:22:47.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:22:47.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:22:47.494 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:22:47.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:22:47.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:22:47.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:22:47.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:22:47.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:22:47.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:22:47.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:22:47.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:22:47.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpstfo06z6:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbriqp69y:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:22:47.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:22:47.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T08:22:47.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: if self._apply_service(spec): 2024-01-13T08:22:47.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T08:22:47.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T08:22:47.495 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T08:22:47.496 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T08:22:47.496 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T08:22:47.496 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T08:22:47.496 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T08:22:47.496 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T08:22:47.496 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T08:22:47.496 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: raise self._value 2024-01-13T08:22:47.496 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T08:22:47.496 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: result = (True, func(*args, **kwds)) 2024-01-13T08:22:47.496 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T08:22:47.496 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: return list(map(*args)) 2024-01-13T08:22:47.496 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:22:47.496 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: return f(*arg) 2024-01-13T08:22:47.496 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:22:47.496 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:22:47.497 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:22:47.497 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: code, '\n'.join(err))) 2024-01-13T08:22:47.497 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:22:47.497 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpstfo06z6:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbriqp69y:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:22:47.497 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:22:47.497 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:22:47.497 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:22:47.497 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new e81c801a-c853-48bd-9922-8a3c006751d8 2024-01-13T08:22:47.497 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:22:47.497 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:22:47.497 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:22:47.497 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:22:47.497 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:22:47.497 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr self.prepare() 2024-01-13T08:22:47.497 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:22:47.498 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:22:47.498 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:22:47.498 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:22:47.498 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:22:47.498 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:22:47.498 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:22:47.498 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:22:47.498 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:22:47.498 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:22:47.498 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:22:47.498 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:22:47.498 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:22:47.498 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:22:47.498 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:22:47.499 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:22:47.499 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:22:47.499 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:22:47.499 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:22:47.499 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:22:47.499 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:22:47.499 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:22:47.499 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:22:47.499 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:22:47.499 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:22:47.499 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:22:47.499 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:22:47.499 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:22:47.499 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:22:47.499 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:22:47.500 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:22:47.500 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:22:47.500 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:22:47.500 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:22:47.500 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:22:47.500 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:22:47.500 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:22:47.500 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:22:47.500 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:22:47.500 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:22:47.500 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:22:47.500 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:22:47.500 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:22:47.501 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:22:47.501 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:22:47.501 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:22:47.501 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:22:47.501 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:22:47.501 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:22:47.501 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:22:47.501 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:22:47.501 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:22:47.501 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:22:47.501 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: main() 2024-01-13T08:22:47.501 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:22:47.501 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: r = ctx.func(ctx) 2024-01-13T08:22:47.501 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:22:47.501 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:22:47.502 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:22:47.502 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:22:47.502 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:22:47.502 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:22:47.502 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:22:47.502 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:22:47.502 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:22:47.502 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:22:47.502 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:22:47.502 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:22:47.502 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpstfo06z6:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbriqp69y:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:22:47.502 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: cluster 2024-01-13T08:22:45.990683+0000 mgr.smithi099.oodult (mgr.14180) 194 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:47.502 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:47 smithi099 conmon[33313]: cluster 2024-01-13T08:22:46.032649+0000 mon.smithi099 (mon.0) 537 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T08:22:47.564 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:22:49.364 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:22:49.386 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:49 smithi099 conmon[33313]: cluster 2024-01-13T08:22:47.991139+0000 mgr.smithi099.oodult (mgr.14180) 195 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:49.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:49 smithi131 conmon[38024]: cluster 2024-01-13T08:22:47.991139+0000 mgr.smithi099.oodult (mgr.14180) 195 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:49.959 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:22:50.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:50 smithi099 conmon[33313]: audit 2024-01-13T08:22:49.367630+0000 mon.smithi099 (mon.0) 538 : audit [DBG] from='client.? 172.21.15.99:0/1443334249' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:22:50.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:50 smithi131 conmon[38024]: audit 2024-01-13T08:22:49.367630+0000 mon.smithi099 (mon.0) 538 : audit [DBG] from='client.? 172.21.15.99:0/1443334249' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:22:50.960 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:22:51.377 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:51 smithi099 conmon[33313]: cluster 2024-01-13T08:22:49.991546+0000 mgr.smithi099.oodult (mgr.14180) 196 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:51.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:51 smithi131 conmon[38024]: cluster 2024-01-13T08:22:49.991546+0000 mgr.smithi099.oodult (mgr.14180) 196 : cluster 2024-01-13T08:22:51.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:51 smithi131 conmon[38024]: [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:52.664 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:22:53.235 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:22:53.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:53 smithi131 conmon[38024]: cluster 2024-01-13T08:22:51.991926+0000 mgr.smithi099.oodult (mgr.14180) 197 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:53.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:53 smithi131 conmon[38024]: 2024-01-13T08:22:53.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:53 smithi131 conmon[38024]: audit 2024-01-13T08:22:52.667362+0000 mon.smithi099 (mon.0) 539 : audit [DBG] from='client.? 172.21.15.99:0/4004111612' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:22:53.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:53 smithi099 conmon[33313]: cluster 2024-01-13T08:22:51.991926+0000 mgr.smithi099.oodult (mgr.14180) 197 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:53.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:53 smithi099 conmon[33313]: audit 2024-01-13T08:22:52.667362+0000 mon.smithi099 (mon.0) 539 : audit [DBG] from='client.? 172.21.15.99:0/4004111612' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:22:54.237 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:22:55.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:55 smithi131 conmon[38024]: cluster 2024-01-13T08:22:53.992377+0000 mgr.smithi099.oodult (mgr.14180) 198 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:55.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:55 smithi131 conmon[38024]: 2024-01-13T08:22:55.437 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:55 smithi099 conmon[33313]: cluster 2024-01-13T08:22:53.992377+0000 mgr.smithi099.oodult (mgr.14180) 198 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:55.994 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:22:56.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:56 smithi131 conmon[38024]: audit 2024-01-13T08:22:55.998564+0000 mon.smithi099 (mon.0) 540 : audit [DBG] 2024-01-13T08:22:56.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:56 smithi131 conmon[38024]: from='client.? 172.21.15.99:0/762879273' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:22:56.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:56 smithi099 conmon[33313]: audit 2024-01-13T08:22:55.998564+0000 mon.smithi099 (mon.0) 540 : audit [DBG] from='client.? 172.21.15.99:0/762879273' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:22:56.568 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:22:57.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:57 smithi131 conmon[38024]: cluster 2024-01-13T08:22:55.992768+0000 mgr.smithi099.oodult (mgr.14180) 199 : 2024-01-13T08:22:57.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:57 smithi131 conmon[38024]: cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:57.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:57 smithi099 conmon[33313]: cluster 2024-01-13T08:22:55.992768+0000 mgr.smithi099.oodult (mgr.14180) 199 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:57.570 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:22:59.308 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:22:59.337 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:59 smithi099 conmon[33313]: cluster 2024-01-13T08:22:57.993310 2024-01-13T08:22:59.337 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:22:59 smithi099 conmon[33313]: +0000 mgr.smithi099.oodult (mgr.14180) 200 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:59.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:22:59 smithi131 conmon[38024]: cluster 2024-01-13T08:22:57.993310+0000 mgr.smithi099.oodult (mgr.14180) 200 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:22:59.841 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:23:00.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:00 smithi131 conmon[38024]: audit 2024-01-13T08:22:59.311947+0000 mon.smithi099 (mon.0) 541 : audit [DBG] from='client.? 172.21.15.99:0/4266189578' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:23:00.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:00 smithi099 conmon[33313]: audit 2024-01-13T08:22:59.311947+0000 mon.smithi099 (mon.0) 541 : audit [DBG] from='client.? 172.21.15.99:0/4266189578' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:23:00.843 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:23:01.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:01 smithi099 conmon[33313]: cluster 2024-01-13T08:22:59.993735+0000 mgr.smithi099.oodult (mgr.14180) 201 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:01.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:01 smithi131 conmon[38024]: cluster 2024-01-13T08:22:59.993735+0000 mgr.smithi099.oodult (mgr.14180) 201 : cluster [DBG] 2024-01-13T08:23:01.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:01 smithi131 conmon[38024]: pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:02.602 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:23:03.150 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:23:03.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:03 smithi131 conmon[38024]: cluster 2024-01-13T08:23:01.994162+0000 mgr.smithi099.oodult (mgr.14180) 202 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:03.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:03 smithi131 conmon[38024]: 2024-01-13T08:23:03.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:03 smithi131 conmon[38024]: audit 2024-01-13T08:23:02.606090+0000 mon.smithi099 (mon.0) 542 : audit [DBG] from='client.? 172.21.15.99:0/1470112068' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:23:03.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:03 smithi099 conmon[33313]: cluster 2024-01-13T08:23:01.994162+0000 mgr.smithi099.oodult (mgr.14180) 202 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:03.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:03 smithi099 conmon[33313]: audit 2024-01-13T08:23:02.606090+0000 mon.smithi099 (mon.0) 542 : audit [DBG] from='client.? 172.21.15.99:0/1470112068' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:23:04.151 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:23:05.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:05 smithi099 conmon[33313]: cluster 2024-01-13T08:23:03.994765+0000 mgr.smithi099.oodult (mgr.14180) 203 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:05.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:05 smithi131 conmon[38024]: cluster 2024-01-13T08:23:03.994765+0000 mgr.smithi099.oodult (mgr.14180) 203 : cluster [DBG] 2024-01-13T08:23:05.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:05 smithi131 conmon[38024]: pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:05.977 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:23:06.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:06 smithi131 conmon[38024]: audit 2024-01-13T08:23:05.981464+0000 mon.smithi099 (mon.0) 543 : audit 2024-01-13T08:23:06.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:06 smithi131 conmon[38024]: [DBG] from='client.? 172.21.15.99:0/2102178401' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:23:06.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:06 smithi099 conmon[33313]: audit 2024-01-13T08:23:05.981464+0000 mon.smithi099 (mon. 2024-01-13T08:23:06.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:06 smithi099 conmon[33313]: 0) 543 : audit [DBG] from='client.? 172.21.15.99:0/2102178401' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:23:06.557 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:23:07.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:07 smithi131 conmon[38024]: cluster 2024-01-13T08:23:05.995232+0000 mgr.smithi099.oodult (mgr.14180) 204 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:07.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:07 smithi099 conmon[33313]: cluster 2024-01-13T08:23:05.995232+0000 mgr.smithi099.oodult (mgr.14180) 204 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:07.559 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:23:09.310 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:23:09.335 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:09 smithi099 conmon[33313]: cluster 2024-01-13T08:23:07.995599+0000 mgr.smithi099.oodult (mgr.14180) 205 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:09.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:09 smithi131 conmon[38024]: cluster 2024-01-13T08:23:07.995599+0000 mgr.smithi099.oodult (mgr.14180) 205 : cluster [DBG] 2024-01-13T08:23:09.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:09 smithi131 conmon[38024]: pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:09.806 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:23:10.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:10 smithi131 conmon[38024]: audit 2024-01-13T08:23:09.314003+0000 mon.smithi099 (mon.0) 544 : audit [DBG] from='client.? 172.21.15.99:0/166392229' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:23:10.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:10 smithi099 conmon[33313]: audit 2024-01-13T08:23:09.314003+0000 mon.smithi099 (mon.0) 544 : audit [DBG] from='client.? 172.21.15.99:0/166392229' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:23:10.808 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:23:11.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:11 smithi131 conmon[38024]: cluster 2024-01-13T08:23:09.996220+0000 mgr.smithi099.oodult (mgr.14180) 206 : cluster [DBG] 2024-01-13T08:23:11.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:11 smithi131 conmon[38024]: pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:11.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:11 smithi099 conmon[33313]: cluster 2024-01-13T08:23:09.996220+0000 mgr.smithi099.oodult (mgr.14180) 206 : cluster [DBG] 2024-01-13T08:23:11.406 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:11 smithi099 conmon[33313]: pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:12.451 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:23:12.964 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:23:13.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:13 smithi099 conmon[33313]: cluster 2024-01-13T08:23:11.996917+0000 mgr.smithi099.oodult (mgr.14180) 207 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:13.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:13 smithi099 conmon[33313]: audit 2024-01-13T08:23:12.455102+0000 mon.smithi099 (mon.0) 545 : audit [DBG] from='client.? 172.21.15.99:0/3254382006' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:23:13.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:13 smithi131 conmon[38024]: cluster 2024-01-13T08:23:11.996917+0000 mgr.smithi099.oodult (mgr.14180) 207 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:13.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:13 smithi131 conmon[38024]: 2024-01-13T08:23:13.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:13 smithi131 conmon[38024]: audit 2024-01-13T08:23:12.455102+0000 mon.smithi099 (mon.0) 545 : audit [DBG] from='client.? 172.21.15.99:0/3254382006' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:23:13.965 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:23:15.359 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:15 smithi099 conmon[33313]: cluster 2024-01-13T08:23:13.997360+0000 mgr.smithi099.oodult (mgr.14180) 208 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:15.359 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:15 smithi099 conmon[33313]: 2024-01-13T08:23:15.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:15 smithi131 conmon[38024]: cluster 2024-01-13T08:23:13.997360+0000 mgr.smithi099.oodult (mgr.14180) 208 : cluster [DBG] 2024-01-13T08:23:15.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:15 smithi131 conmon[38024]: pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:15.664 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:23:16.221 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:23:16.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:16 smithi099 conmon[33313]: audit 2024-01-13T08:23:15.668486+0000 mon.smithi099 (mon.0) 546 : audit [DBG] from='client.? 172.21.15.99:0/1699467560' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:23:16.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:16 smithi131 conmon[38024]: audit 2024-01-13T08:23:15.668486+0000 mon.smithi099 (mon.0) 546 : audit [DBG] 2024-01-13T08:23:16.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:16 smithi131 conmon[38024]: from='client.? 172.21.15.99:0/1699467560' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:23:17.222 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:23:17.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:17 smithi099 conmon[33313]: cluster 2024-01-13T08:23:15.997992+0000 mgr.smithi099.oodult (mgr.14180) 209 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:17.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:17 smithi131 conmon[38024]: cluster 2024-01-13T08:23:15.997992+0000 mgr.smithi099.oodult (mgr.14180) 209 : cluster 2024-01-13T08:23:17.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:17 smithi131 conmon[38024]: [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:18.972 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:23:19.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:19 smithi099 conmon[33313]: cluster 2024-01-13T08:23:17.998378+0000 mgr.smithi099.oodult (mgr.14180) 210 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:19.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:19 smithi099 conmon[33313]: audit 2024-01-13T08:23:18.976056+0000 mon.smithi099 (mon.0) 547 : audit [DBG] from='client.? 172.21.15.99:0/1629206284' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:23:19.536 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:23:19.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:19 smithi131 conmon[38024]: cluster 2024-01-13T08:23:17.998378+0000 mgr.smithi099.oodult (mgr.14180) 210 : cluster [DBG] 2024-01-13T08:23:19.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:19 smithi131 conmon[38024]: pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:19.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:19 smithi131 conmon[38024]: audit 2024-01-13T08:23:18.976056+0000 mon.smithi099 (mon.0) 547 : audit [DBG] from='client.? 172.21.15.99:0/1629206284' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:23:20.538 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:23:21.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:21 smithi099 conmon[33313]: cluster 2024-01-13T08:23:19.998802+0000 mgr.smithi099.oodult (mgr.14180) 211 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:21.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:21 smithi131 conmon[38024]: cluster 2024-01-13T08:23:19.998802+0000 mgr.smithi099.oodult (mgr.14180) 211 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:22.290 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:23:22.878 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:23:23.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:23 smithi099 conmon[33313]: cluster 2024-01-13T08:23:21.999247+0000 mgr.smithi099.oodult (mgr.14180) 212 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:23.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:23 smithi099 conmon[33313]: audit 2024-01-13T08:23:22.293842+0000 mon.smithi099 (mon.0) 548 : audit [DBG] from='client.? 172.21.15.99:0/2798548846' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:23:23.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:23 smithi131 conmon[38024]: cluster 2024-01-13T08:23:21.999247+0000 mgr.smithi099.oodult (mgr.14180) 212 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:23.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:23 smithi131 conmon[38024]: audit 2024-01-13T08:23:22.293842+0000 mon.smithi099 (mon.0) 548 : audit [DBG] from='client.? 172.21.15.99:0/2798548846' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:23:23.880 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:23:25.290 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:25 smithi099 conmon[33313]: cluster 2024-01-13T08:23:23.999643+0000 mgr.smithi099.oodult (mgr.14180) 213 2024-01-13T08:23:25.290 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:25 smithi099 conmon[33313]: : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:25.587 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:23:25.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:25 smithi131 conmon[38024]: cluster 2024-01-13T08:23:23.999643+0000 mgr.smithi099.oodult (mgr.14180) 213 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:26.120 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:23:26.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:26 smithi099 conmon[33313]: audit 2024-01-13T08:23:25.590991+0000 mon.smithi099 (mon.0) 549 : audit [DBG] from='client.? 172.21.15.99:0/3038775274' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:23:26.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:26 smithi131 conmon[38024]: audit 2024-01-13T08:23:25.590991+0000 mon.smithi099 (mon.0) 549 : audit [DBG] from='client.? 172.21.15.99:0/3038775274' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:23:27.122 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:23:27.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:27 smithi099 conmon[33313]: cluster 2024-01-13T08:23:26.000098+0000 mgr.smithi099.oodult (mgr.14180) 214 : cluster 2024-01-13T08:23:27.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:27 smithi099 conmon[33313]: [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:27.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:27 smithi131 conmon[38024]: cluster 2024-01-13T08:23:26.000098+0000 mgr.smithi099.oodult (mgr.14180) 214 : cluster 2024-01-13T08:23:27.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:27 smithi131 conmon[38024]: [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:28.831 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:23:29.351 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:23:29.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:29 smithi099 conmon[33313]: cluster 2024-01-13T08:23:28.000511+0000 mgr.smithi099.oodult (mgr.14180) 215 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:29.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:29 smithi099 conmon[33313]: audit 2024-01-13T08:23:28.835296+0000 mon.smithi099 (mon 2024-01-13T08:23:29.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:29 smithi099 conmon[33313]: .0) 550 : audit [DBG] from='client.? 172.21.15.99:0/941559093' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:23:29.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:29 smithi131 conmon[38024]: cluster 2024-01-13T08:23:28.000511+0000 mgr.smithi099.oodult (mgr.14180) 215 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:29.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:29 smithi131 conmon[38024]: audit 2024-01-13T08:23:28.835296+0000 mon.smithi099 (mon.0) 550 : audit [DBG] from='client.? 172.21.15.99:0/941559093' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:23:30.352 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:23:31.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:31 smithi099 conmon[33313]: cluster 2024-01-13T08:23:30.001062+0000 mgr.smithi099.oodult (mgr.14180) 216 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:31.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:31 smithi131 conmon[38024]: cluster 2024-01-13T08:23:30.001062+0000 mgr.smithi099.oodult (mgr.14180) 216 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:32.087 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:23:32.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:32 smithi099 conmon[33313]: audit 2024-01-13T08:23:32.091247+0000 mon.smithi099 (mon.0) 551 : audit [DBG] from='client.? 172.21.15.99:0/4135665031' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:23:32.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:32 smithi131 conmon[38024]: audit 2024-01-13T08:23:32.091247+0000 mon.smithi099 (mon.0 2024-01-13T08:23:32.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:32 smithi131 conmon[38024]: ) 551 : audit [DBG] from='client.? 172.21.15.99:0/4135665031' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:23:32.651 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:23:33.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:33 smithi099 conmon[33313]: cluster 2024-01-13T08:23:32.001479+0000 mgr.smithi099.oodult (mgr.14180) 217 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:33.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:33 smithi131 conmon[38024]: cluster 2024-01-13T08:23:32.001479+0000 mgr.smithi099.oodult (mgr.14180) 217 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:33.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:33 smithi131 conmon[38024]: 2024-01-13T08:23:33.652 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:23:35.393 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:23:35.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:35 smithi099 conmon[33313]: cluster 2024-01-13T08:23:34.002025+0000 mgr.smithi099.oodult (mgr.14180) 218 : cluster [DBG] 2024-01-13T08:23:35.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:35 smithi099 conmon[33313]: pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:35.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:35 smithi131 conmon[38024]: cluster 2024-01-13T08:23:34.002025+0000 mgr.smithi099.oodult (mgr.14180) 218 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:36.025 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:23:36.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:36 smithi099 conmon[33313]: audit 2024-01-13T08:23:35.397508+0000 mon.smithi099 (mon.0) 552 : audit [DBG] from='client.? 172.21.15.99:0/713806407' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:23:36.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:36 smithi131 conmon[38024]: audit 2024-01-13T08:23:35.397508+0000 mon.smithi099 (mon.0) 552 : audit [DBG] from='client.? 172.21.15.99:0/713806407' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:23:37.026 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:23:37.475 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:37 smithi099 conmon[33313]: cluster 2024-01-13T08:23:36.002677+0000 mgr.smithi099.oodult (mgr.14180) 219 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:37.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:37 smithi131 conmon[38024]: cluster 2024-01-13T08:23:36.002677+0000 mgr.smithi099.oodult (mgr.14180) 219 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:38.792 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:23:39.349 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:23:39.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:39 smithi099 conmon[33313]: cluster 2024-01-13T08:23:38.003261+0000 mgr.smithi099.oodult (mgr.14180) 220 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:39.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:39 smithi099 conmon[33313]: audit 2024-01-13T08:23:38.796176+0000 mon.smithi099 (mon.0) 553 : audit [DBG] from='client.? 172.21.15.99:0/3959995344' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:23:39.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:39 smithi131 conmon[38024]: cluster 2024-01-13T08:23:38.003261+0000 mgr.smithi099.oodult (mgr.14180) 220 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:39.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:39 smithi131 conmon[38024]: audit 2024-01-13T08:23:38.796176+0000 mon.smithi099 (mon.0) 553 2024-01-13T08:23:39.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:39 smithi131 conmon[38024]: : audit [DBG] from='client.? 172.21.15.99:0/3959995344' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:23:40.349 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:23:41.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:41 smithi099 conmon[33313]: cluster 2024-01-13T08:23:40.003714+0000 mgr.smithi099.oodult (mgr.14180) 221 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:41.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:41 smithi131 conmon[38024]: cluster 2024-01-13T08:23:40.003714+0000 mgr.smithi099.oodult (mgr.14180) 221 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:42.125 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:23:42.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:42 smithi099 conmon[33313]: audit 2024-01-13T08:23:42.129316+0000 mon.smithi099 (mon.0) 554 : 2024-01-13T08:23:42.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:42 smithi099 conmon[33313]: audit [DBG] from='client.? 172.21.15.99:0/605645202' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:23:42.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:42 smithi131 conmon[38024]: audit 2024-01-13T08:23:42.129316+0000 mon.smithi099 (mon.0) 554 : audit [DBG] from='client.? 172.21.15.99:0/605645202' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:23:42.705 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:23:43.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:43 smithi131 conmon[38024]: cluster 2024-01-13T08:23:42.004095+0000 mgr.smithi099.oodult (mgr.14180) 222 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:43.707 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:23:43.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:43 smithi099 conmon[33313]: cluster 2024-01-13T08:23:42.004095+0000 mgr.smithi099.oodult (mgr.14180) 222 : cluster [DBG] 2024-01-13T08:23:43.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:43 smithi099 conmon[33313]: pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:45.419 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:23:45.598 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:45 smithi099 conmon[33313]: cluster 2024-01-13T08:23:44.004437+0000 mgr.smithi099.oodult (mgr.14180) 223 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:45.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:45 smithi131 conmon[38024]: cluster 2024-01-13T08:23:44.004437+0000 mgr.smithi099.oodult (mgr.14180) 223 : cluster [DBG] 2024-01-13T08:23:45.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:45 smithi131 conmon[38024]: pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:45.956 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:23:46.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:46 smithi131 conmon[38024]: audit 2024-01-13T08:23:45.423164+0000 mon.smithi099 (mon.0) 555 2024-01-13T08:23:46.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:46 smithi131 conmon[38024]: : audit [DBG] from='client.? 172.21.15.99:0/361505884' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:23:46.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:46 smithi131 conmon[38024]: audit 2024-01-13T08:23:45.991908+0000 mon.smithi099 (mon.0) 556 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:23:46.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:46 smithi099 conmon[33313]: audit 2024-01-13T08:23:45.423164+0000 mon.smithi099 (mon 2024-01-13T08:23:46.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:46 smithi099 conmon[33313]: .0) 555 : audit [DBG] from='client.? 172.21.15.99:0/361505884' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:23:46.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:46 smithi099 conmon[33313]: audit 2024-01-13T08:23:45.991908+0000 mon.smithi099 (mon.0) 556 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:23:46.958 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:23:47.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:47 smithi099 conmon[33313]: cluster 2024-01-13T08:23:46.004982+0000 mgr.smithi099.oodult (mgr.14180) 224 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:47.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:47 smithi099 conmon[33313]: audit 2024-01-13T08:23:46.314385+0000 mon.smithi099 (mon.0) 557 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:23:47.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:47 smithi099 conmon[33313]: audit 2024-01-13T08:23:46.314993+0000 mon.smithi099 (mon.0) 558 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T08:23:47.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:47 smithi099 conmon[33313]: audit 2024-01-13T08:23:46.318988+0000 mon.smithi099 (mon.0) 559 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:23:47.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:47 smithi099 conmon[33313]: cluster 2024-01-13T08:23:46.319311+0000 mgr.smithi099.oodult (mgr.14180) 225 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:47.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:47 smithi099 conmon[33313]: audit 2024-01-13T08:23:46.322438+0000 mon.smithi099 (mon.0) 560 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T08:23:47.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:47 smithi099 conmon[33313]: audit 2024-01-13T08:23:46 2024-01-13T08:23:47.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:47 smithi099 conmon[33313]: .324980+0000 mon.smithi099 (mon.0) 561 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:23:47.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:47 smithi099 conmon[33313]: audit 2024-01-13T08:23:46.325959+0000 mon.smithi099 (mon.0) 562 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:23:47.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:47 smithi099 conmon[33313]: audit 2024-01-13T08:23:46.326119+0000 mon.smithi099 (mon.0) 563 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:23:47.482 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:47 smithi099 conmon[33313]: audit 2024-01-13T08:23:46.326523+0000 mon.smithi099 (mon.0) 564 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:23:47.537 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:47 smithi131 conmon[38024]: cluster 2024-01-13T08:23:46.004982+0000 mgr.smithi099.oodult (mgr.14180) 224 : cluster [DBG] 2024-01-13T08:23:47.537 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:47 smithi131 conmon[38024]: pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:47.537 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:47 smithi131 conmon[38024]: audit 2024-01-13T08:23:46.314385+0000 mon.smithi099 (mon.0) 557 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:23:47.537 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:47 smithi131 conmon[38024]: audit 2024-01-13T08:23:46.314993+0000 mon.smithi099 (mon.0) 558 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T08:23:47.537 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:47 smithi131 conmon[38024]: audit 2024-01-13T08:23:46.318988+0000 mon.smithi099 (mon.0) 559 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:23:47.537 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:47 smithi131 conmon[38024]: cluster 2024-01-13T08:23:46.319311+0000 mgr.smithi099.oodult (mgr.14180) 225 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:47.537 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:47 smithi131 conmon[38024]: audit 2024-01-13T08:23:46.322438+0000 mon.smithi099 (mon.0) 560 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T08:23:47.537 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:47 smithi131 conmon[38024]: audit 2024-01-13T08:23:46.324980+0000 mon.smithi099 (mon.0) 561 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:23:47.537 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:47 smithi131 conmon[38024]: audit 2024-01-13T08:23:46.325959+0000 mon.smithi099 (mon.0) 562 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:23:47.538 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:47 smithi131 conmon[38024]: audit 2024-01-13T08:23:46.326119+0000 mon.smithi099 (mon.0) 563 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:23:47.538 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:47 smithi131 conmon[38024]: audit 2024-01-13T08:23:46.326523+0000 mon.smithi099 (mon.0) 564 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:23:48.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:48 smithi131 conmon[38024]: cluster 2024-01-13T08:23:47.317372+0000 mon.smithi099 (mon.0) 565 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T08:23:48.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:48 smithi131 conmon[38024]: cluster 2024-01-13T08:23:47.317398+0000 mon.smithi099 (mon.0) 566 : cluster [INF] Cluster is now healthy 2024-01-13T08:23:48.669 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:48 smithi099 conmon[33313]: cluster 2024-01-13T08:23:47.317372+0000 mon.smithi099 (mon.0) 565 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T08:23:48.669 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:48 smithi099 conmon[33313]: cluster 2024-01-13T08:23:47.317398+0000 mon.smithi099 (mon.0) 566 : cluster [INF] Cluster is now healthy 2024-01-13T08:23:49.279 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:23:49.521 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:49 smithi099 conmon[33313]: cluster 2024-01-13T08:23:48.319598+0000 mgr.smithi099.oodult (mgr.14180) 226 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:49.521 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:49 smithi099 conmon[33313]: 2024-01-13T08:23:49.521 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:49 smithi099 conmon[33313]: audit 2024-01-13T08:23:48.849398+0000 mon.smithi131 (mon.1) 12 : audit [INF] from='client.? 172.21.15.131:0/1006654437' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11f5f35c-81e4-4555-9a25-a75daadf41cd"}]: dispatch 2024-01-13T08:23:49.522 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:49 smithi099 conmon[33313]: audit 2024-01-13T08:23:48.849737+0000 mon.smithi099 (mon.0) 567 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11f5f35c-81e4-4555-9a25-a75daadf41cd"}]: dispatch 2024-01-13T08:23:49.522 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:49 smithi099 conmon[33313]: audit 2024-01-13T08:23:48.852962+0000 mon.smithi099 (mon.0) 568 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "11f5f35c-81e4-4555-9a25-a75daadf41cd"}]': finished 2024-01-13T08:23:49.522 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:49 smithi099 conmon[33313]: cluster 2024-01- 2024-01-13T08:23:49.522 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:49 smithi099 conmon[33313]: 13T08:23:48.853034+0000 mon.smithi099 (mon. 2024-01-13T08:23:49.522 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:49 smithi099 conmon[33313]: 0) 569 : cluster [DBG] osdmap e26: 1 total, 0 up, 1 in 2024-01-13T08:23:49.522 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:49 smithi099 conmon[33313]: audit 2024-01-13T08:23:48. 2024-01-13T08:23:49.522 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:49 smithi099 conmon[33313]: 853146+0000 mon.smithi099 (mon.0) 570 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:23:49.522 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:49 smithi099 conmon[33313]: audit 2024-01-13T08:23:49.120698+0000 2024-01-13T08:23:49.522 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:49 smithi099 conmon[33313]: mon.smithi099 (mon.0) 571 : audit [INF] from='client.? 172.21.15.99:0/3471113236' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f9c1a26-e2e9-4648-be87-29b5ac68c198"}]: dispatch 2024-01-13T08:23:49.522 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:49 smithi099 conmon[33313]: audit 2024-01-13T08:23:49.124205+0000 mon.smithi099 2024-01-13T08:23:49.522 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:49 smithi099 conmon[33313]: (mon.0) 572 : audit [INF] from='client.? 172.21.15.99:0/3471113236' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2f9c1a26-e2e9-4648-be87-29b5ac68c198"}]': finished 2024-01-13T08:23:49.522 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:49 smithi099 conmon[33313]: cluster 2024-01-13T08:23:49.522 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:49 smithi099 conmon[33313]: 2024-01-13T08:23:49.124252+0000 mon.smithi099 (mon.0) 573 : cluster [DBG] 2024-01-13T08:23:49.522 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:49 smithi099 conmon[33313]: osdmap e27: 2 total, 0 up, 2 in 2024-01-13T08:23:49.523 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:49 smithi099 conmon[33313]: audit 2024-01-13T08:23 2024-01-13T08:23:49.523 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:49 smithi099 conmon[33313]: :49.124367+0000 mon.smithi099 (mon.0) 574 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:23:49.523 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:49 smithi099 conmon[33313]: audit 2024-01-13T08:23:49.124674+0000 mon.smithi099 (mon.0) 2024-01-13T08:23:49.523 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:49 smithi099 conmon[33313]: 575 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-13T08:23:49.523 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:49 smithi099 conmon[33313]: audit 2024-01-13T08:23:49.523 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:49 smithi099 conmon[33313]: 2024-01-13T08:23:49.140011+0000 mon.smithi131 (mon.1) 13 : audit [INF] from='client.? 172.21.15.131:0/4257040070' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:23:49.523 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:49 smithi099 conmon[33313]: audit 2024-01-13T08:23:49.140404+0000 mon.smithi099 (mon.0) 576 : 2024-01-13T08:23:49.523 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:49 smithi099 conmon[33313]: audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:23:49.523 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:49 smithi099 conmon[33313]: audit 2024-01-13T08:23:49.145342+0000 mon.smithi099 (mon.0 2024-01-13T08:23:49.523 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:49 smithi099 conmon[33313]: ) 577 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:23:49.523 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:49 smithi099 conmon[33313]: cluster 2024- 2024-01-13T08:23:49.523 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:49 smithi099 conmon[33313]: 01-13T08:23:49.145425+0000 mon.smithi099 (mon.0) 578 : cluster [DBG] osdmap e28: 1 total, 0 up, 1 in 2024-01-13T08:23:49.523 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:49 smithi099 conmon[33313]: audit 2024-01-13T08:23:49.145531+0000 mon.smithi099 (mon.0) 579 2024-01-13T08:23:49.523 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:49 smithi099 conmon[33313]: : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-13T08:23:49.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:49 smithi131 conmon[38024]: cluster 2024-01-13T08:23:48.319598+0000 mgr.smithi099.oodult (mgr.14180) 2024-01-13T08:23:49.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:49 smithi131 conmon[38024]: 226 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:49.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:49 smithi131 conmon[38024]: audit 2024-01-13T08:23:48.849398+0000 mon.smithi131 (mon.1) 12 : audit [INF] from='client.? 172.21.15.131:0/1006654437' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11f5f35c-81e4-4555-9a25-a75daadf41cd"}]: dispatch 2024-01-13T08:23:49.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:49 smithi131 conmon[38024]: audit 2024-01-13T08:23:48.849737+0000 mon.smithi099 (mon.0) 567 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11f5f35c-81e4-4555-9a25-a75daadf41cd"}]: dispatch 2024-01-13T08:23:49.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:49 smithi131 conmon[38024]: audit 2024-01-13T08:23:48.852962+0000 mon.smithi099 (mon.0) 568 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "11f5f35c-81e4-4555-9a25-a75daadf41cd"}]': finished 2024-01-13T08:23:49.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:49 smithi131 conmon[38024]: cluster 2024-01-13T08:23:48.853034+0000 mon.smithi099 2024-01-13T08:23:49.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:49 smithi131 conmon[38024]: (mon.0) 569 : cluster [DBG] osdmap e26: 1 total, 0 up, 1 in 2024-01-13T08:23:49.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:49 smithi131 conmon[38024]: audit 2024-01-13T08:23:48.853146+0000 mon.smithi099 (mon.0) 570 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:23:49.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:49 smithi131 conmon[38024]: audit 2024-01-13T08:23:49.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:49 smithi131 conmon[38024]: 2024-01-13T08:23:49.120698+0000 mon.smithi099 (mon.0) 571 : audit [INF] from='client.? 172.21.15.99:0/3471113236' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f9c1a26-e2e9-4648-be87-29b5ac68c198"}]: dispatch 2024-01-13T08:23:49.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:49 smithi131 conmon[38024]: audit 2024-01-13T08:23:49.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:49 smithi131 conmon[38024]: 2024-01-13T08:23:49.124205+0000 mon.smithi099 (mon.0) 572 : audit [INF] from='client.? 172.21.15.99:0/3471113236' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2f9c1a26-e2e9-4648-be87-29b5ac68c198"}]': finished 2024-01-13T08:23:49.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:49 smithi131 conmon[38024]: cluster 2024-01-13T08:23:49.124252+0000 2024-01-13T08:23:49.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:49 smithi131 conmon[38024]: mon.smithi099 (mon.0) 573 : cluster [DBG] osdmap e27: 2 total, 0 up, 2 in 2024-01-13T08:23:49.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:49 smithi131 conmon[38024]: audit 2024-01-13T08:23:49.124367+0000 mon.smithi099 ( 2024-01-13T08:23:49.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:49 smithi131 conmon[38024]: mon.0) 574 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:23:49.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:49 smithi131 conmon[38024]: audit 2024-01 2024-01-13T08:23:49.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:49 smithi131 conmon[38024]: -13T08:23:49.124674+0000 mon.smithi099 (mon.0) 575 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-13T08:23:49.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:49 smithi131 conmon[38024]: audit 2024-01-13T08:23:49.140011+0000 mon.smithi131 (mon.1) 2024-01-13T08:23:49.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:49 smithi131 conmon[38024]: 13 : audit [INF] from='client.? 172.21.15.131:0/4257040070' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:23:49.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:49 smithi131 conmon[38024]: audit 2024 2024-01-13T08:23:49.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:49 smithi131 conmon[38024]: -01-13T08:23:49.140404+0000 mon.smithi099 (mon.0) 576 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:23:49.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:49 smithi131 conmon[38024]: audit 2024-01-13T08:23:49.145342+0000 2024-01-13T08:23:49.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:49 smithi131 conmon[38024]: mon.smithi099 (mon.0) 577 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:23:49.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:49 smithi131 conmon[38024]: cluster 2024-01-13T08:23:49.145425+0000 mon.smithi099 (mon. 2024-01-13T08:23:49.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:49 smithi131 conmon[38024]: 0) 578 : cluster [DBG] osdmap e28: 1 total, 0 up, 1 in 2024-01-13T08:23:49.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:49 smithi131 conmon[38024]: audit 2024 2024-01-13T08:23:49.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:49 smithi131 conmon[38024]: -01-13T08:23:49.145531+0000 mon.smithi099 (mon.0) 579 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-13T08:23:49.924 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":28,"num_osds":1,"num_up_osds":0,"osd_up_since":0,"num_in_osds":1,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:23:50.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:50 smithi131 conmon[38024]: audit 2024-01-13T08:23:49.283557+0000 mon.smithi099 (mon.0) 580 : audit [DBG] from='client.? 172.21.15.99:0/1496601086' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:23:50.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:50 smithi131 conmon[38024]: audit 2024-01-13T08:23:49.429465+0000 mon.smithi099 2024-01-13T08:23:50.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:50 smithi131 conmon[38024]: (mon.0) 581 : audit [INF] from='client.? 172.21.15.99:0/503308801' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:23:50.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:50 smithi131 conmon[38024]: audit 2024-01-13T08:23:49.433669+0000 mon.smithi099 (mon.0) 582 : audit [INF] from='client.? 172.21.15.99:0/503308801' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:23:50.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:50 smithi131 conmon[38024]: cluster 2024-01-13T08:23:49.433706+0000 mon.smithi099 (mon.0) 583 : cluster [DBG] osdmap e29: 0 total, 0 up, 0 in 2024-01-13T08:23:50.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:50 smithi099 conmon[33313]: audit 2024-01-13T08:23:49.283557+0000 mon.smithi099 (mon.0) 580 2024-01-13T08:23:50.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:50 smithi099 conmon[33313]: : audit [DBG] from='client.? 172.21.15.99:0/1496601086' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:23:50.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:50 smithi099 conmon[33313]: audit 2024-01-13T08:23:49.429465+0000 mon.smithi099 (mon.0) 581 : audit [INF] from='client.? 172.21.15.99:0/503308801' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:23:50.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:50 smithi099 conmon[33313]: audit 2024-01-13T08:23:49.433669+0000 mon.smithi099 (mon.0) 582 : audit [INF] from='client.? 172.21.15.99:0/503308801' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 1, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:23:50.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:50 smithi099 conmon[33313]: cluster 2024-01-13T08:23:49.433706+0000 mon.smithi099 (mon.0) 583 : cluster [DBG] osdmap e29: 0 total, 0 up, 0 in 2024-01-13T08:23:50.925 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:23:51.541 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: cephadm 2024-01-13T08:23:49.878903+0000 mgr.smithi099.oodult (mgr.14180) 227 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi131', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:23:51.541 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:23:51.541 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:23:51.541 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: return f(*arg) 2024-01-13T08:23:51.541 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:23:51.541 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:23:51.542 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:23:51.542 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: code, '\n'.join(err))) 2024-01-13T08:23:51.542 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:23:51.542 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpm3lbpqlt:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpl_srskaa:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:23:51.542 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:23:51.542 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:23:51.542 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:23:51.542 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 11f5f35c-81e4-4555-9a25-a75daadf41cd 2024-01-13T08:23:51.542 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:23:51.542 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:23:51.542 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:23:51.542 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:23:51.542 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:23:51.542 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr self.prepare() 2024-01-13T08:23:51.542 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:23:51.543 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:23:51.543 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:23:51.543 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:23:51.543 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:23:51.543 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:23:51.543 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:23:51.543 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:23:51.543 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:23:51.543 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:23:51.543 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:23:51.544 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:23:51.544 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:23:51.544 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:23:51.544 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:23:51.544 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:23:51.544 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:23:51.544 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:23:51.544 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:23:51.544 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:23:51.544 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:23:51.545 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:23:51.545 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:23:51.545 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:23:51.545 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:23:51.545 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:23:51.545 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:23:51.545 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:23:51.545 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:23:51.545 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:23:51.545 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:23:51.545 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:23:51.545 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:23:51.545 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:23:51.545 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:23:51.547 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:23:51.547 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:23:51.547 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:23:51.548 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:23:51.548 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:23:51.548 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:23:51.548 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:23:51.548 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:23:51.548 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:23:51.548 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:23:51.549 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:23:51.549 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:23:51.549 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:23:51.549 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:23:51.549 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:23:51.549 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:23:51.549 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:23:51.549 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: main() 2024-01-13T08:23:51.549 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:23:51.549 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: r = ctx.func(ctx) 2024-01-13T08:23:51.549 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:23:51.549 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:23:51.549 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:23:51.549 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:23:51.550 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:23:51.550 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:23:51.550 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-8777 2024-01-13T08:23:51.550 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: 4f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:23:51.550 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:23:51.550 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:23:51.550 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:23:51.550 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:23:51.550 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:23:51.550 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpm3lbpqlt:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpl_srskaa:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:23:51.550 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: cluster 2024-01-13T08:23:50.319867+0000 mgr.smithi099.oodult (mgr.14180) 228 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:51.550 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: cephadm 2024-01-13T08:23:50.341961+0000 mgr.smithi099.oodult (mgr.14180) 229 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi131', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:23:51.550 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:23:51.550 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:23:51.550 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: return f(*arg) 2024-01-13T08:23:51.551 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:23:51.551 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:23:51.551 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:23:51.551 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: code, '\n'.join(err))) 2024-01-13T08:23:51.551 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi099/config 2024-01-13T08:23:51.551 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpuxber58i:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp847v5n03:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:23:51.551 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:23:51.551 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:23:51.551 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:23:51.551 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 2f9c1a26-e2e9-4648-be87-29b5ac68c198 2024-01-13T08:23:51.551 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:23:51.551 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 --yes-i-really-mean-it 2024-01-13T08:23:51.551 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr stderr: purged osd.1 2024-01-13T08:23:51.551 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:23:51.552 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:23:51.552 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr self.prepare() 2024-01-13T08:23:51.552 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:23:51.552 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:23:51.552 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:23:51.552 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:23:51.552 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:23:51.552 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:23:51.552 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:23:51.552 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:23:51.552 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:23:51.552 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:23:51.553 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:23:51.553 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:23:51.553 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:23:51.553 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:23:51.553 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:23:51.553 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:23:51.553 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:23:51.553 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:23:51.553 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:23:51.553 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:23:51.553 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:23:51.553 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:23:51.553 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:23:51.553 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:23:51.553 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:23:51.554 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:23:51.554 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:23:51.554 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:23:51.554 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:23:51.554 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:23:51.554 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:23:51.554 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:23:51.554 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:23:51.554 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:23:51.554 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:23:51.554 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:23:51.554 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:23:51.554 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:23:51.554 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr rol 2024-01-13T08:23:51.555 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: lback_osd(self.args, self.osd_id) 2024-01-13T08:23:51.555 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:23:51.555 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:23:51.555 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:23:51.555 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:23:51.555 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:23:51.555 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:23:51.555 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:23:51.555 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:23:51.555 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:23:51.555 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:23:51.555 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 1 2024-01-13T08:23:51.556 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:23:51.556 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:23:51.556 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: main() 2024-01-13T08:23:51.556 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:23:51.556 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: r = ctx.func(ctx) 2024-01-13T08:23:51.556 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:23:51.556 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:23:51.556 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:23:51.556 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:23:51.556 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:23:51.556 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:23:51.556 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:23:51.556 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:23:51.557 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:23:51.557 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:23:51.557 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:23:51.557 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:23:51.557 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpuxber58i:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp847v5n03:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:23:51.557 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: cephadm 2024-01-13T08:23:50.343229+0000 mgr.smithi099.oodult (mgr.14180) 230 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T08:23:51.557 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: service_id: all-available-devices 2024-01-13T08:23:51.557 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: service_name: osd.all-available-devices 2024-01-13T08:23:51.557 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: placement: 2024-01-13T08:23:51.557 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: host_pattern: '*' 2024-01-13T08:23:51.557 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: spec: 2024-01-13T08:23:51.557 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: data_devices: 2024-01-13T08:23:51.557 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: all: true 2024-01-13T08:23:51.557 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: filter_logic: AND 2024-01-13T08:23:51.558 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: objectstore: bluestore 2024-01-13T08:23:51.558 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:23:51.558 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpm3lbpqlt:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpl_srskaa:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:23:51.558 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:23:51.558 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:23:51.558 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:23:51.558 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 11f5f35c-81e4-4555-9a25-a75daadf41cd 2024-01-13T08:23:51.558 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:23:51.558 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:23:51.558 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:23:51.559 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:23:51.559 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:23:51.559 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr self.prepare() 2024-01-13T08:23:51.559 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:23:51.559 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:23:51.559 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:23:51.559 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:23:51.559 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:23:51.559 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:23:51.559 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:23:51.559 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:23:51.560 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:23:51.560 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:23:51.560 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:23:51.560 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:23:51.560 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:23:51.560 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:23:51.560 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:23:51.560 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:23:51.560 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:23:51.560 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:23:51.560 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:23:51.560 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:23:51.560 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:23:51.560 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", 2024-01-13T08:23:51.561 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: line 194, in dispatch 2024-01-13T08:23:51.561 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:23:51.561 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:23:51.561 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:23:51.561 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:23:51.561 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:23:51.561 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:23:51.561 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:23:51.561 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:23:51.561 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:23:51.561 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:23:51.561 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:23:51.561 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:23:51.561 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:23:51.561 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:23:51.562 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:23:51.562 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:23:51.562 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:23:51.562 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:23:51.562 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:23:51.562 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:23:51.562 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:23:51.562 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:23:51.562 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:23:51.562 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:23:51.562 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:23:51.562 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:23:51.562 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:23:51.562 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:23:51.562 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:23:51.563 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:23:51.563 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: main() 2024-01-13T08:23:51.563 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:23:51.563 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: r = ctx.func(ctx) 2024-01-13T08:23:51.563 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:23:51.563 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:23:51.563 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:23:51.563 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:23:51.563 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:23:51.563 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:23:51.563 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:23:51.563 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:23:51.563 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:23:51.563 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:23:51.563 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:23:51.564 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:23:51.564 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpm3lbpqlt:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpl_srskaa:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:23:51.564 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:23:51.564 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T08:23:51.564 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: if self._apply_service(spec): 2024-01-13T08:23:51.564 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T08:23:51.564 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T08:23:51.564 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T08:23:51.564 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T08:23:51.564 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T08:23:51.564 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T08:23:51.564 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T08:23:51.564 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T08:23:51.565 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T08:23:51.565 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: raise self._value 2024-01-13T08:23:51.565 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T08:23:51.565 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: result = (True, func(*args, **kwds)) 2024-01-13T08:23:51.565 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T08:23:51.565 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: return list(map(*args)) 2024-01-13T08:23:51.565 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:23:51.565 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: return f(*arg) 2024-01-13T08:23:51.566 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:23:51.566 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:23:51.566 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:23:51.566 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: code, '\n'.join(err))) 2024-01-13T08:23:51.566 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:23:51.566 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpm3lbpqlt:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpl_srskaa:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:23:51.566 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:23:51.566 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:23:51.566 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:23:51.566 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name clie 2024-01-13T08:23:51.566 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: nt.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 11f5f35c-81e4-4555-9a25-a75daadf41cd 2024-01-13T08:23:51.566 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:23:51.566 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:23:51.566 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:23:51.567 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:23:51.567 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:23:51.567 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr self.prepare() 2024-01-13T08:23:51.567 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:23:51.567 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:23:51.567 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:23:51.567 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:23:51.567 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:23:51.567 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:23:51.567 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:23:51.567 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:23:51.567 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:23:51.567 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:23:51.568 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:23:51.568 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:23:51.568 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:23:51.568 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:23:51.568 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:23:51.568 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:23:51.568 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:23:51.568 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:23:51.568 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:23:51.568 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:23:51.568 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:23:51.568 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:23:51.569 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:23:51.569 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:23:51.569 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:23:51.569 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:23:51.569 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:23:51.569 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:23:51.569 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:23:51.569 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:23:51.569 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:23:51.569 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:23:51.569 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:23:51.569 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:23:51.569 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:23:51.570 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:23:51.570 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:23:51.570 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:23:51.570 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:23:51.570 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:23:51.570 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:23:51.570 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:23:51.570 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:23:51.570 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:23:51.570 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:23:51.570 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:23:51.570 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:23:51.570 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:23:51.570 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:23:51.570 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:23:51.571 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:23:51.571 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:23:51.571 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: main() 2024-01-13T08:23:51.571 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:23:51.571 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: r = ctx.func(ctx) 2024-01-13T08:23:51.571 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:23:51.571 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:23:51.571 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:23:51.571 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:23:51.571 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:23:51.571 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:23:51.571 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:23:51.571 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:23:51.571 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:23:51.572 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:23:51.572 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:23:51.572 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:23:51.572 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpm3lbpqlt:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpl_srskaa:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:23:51.572 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:51 smithi099 conmon[33313]: cluster 2024-01-13T08:23:50.343819+0000 mgr.smithi099.oodult (mgr.14180) 231 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:51.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: cephadm 2024-01-13T08:23:49.878903+0000 mgr.smithi099.oodult (mgr.14180) 227 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi131', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:23:51.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:23:51.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:23:51.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: return f(*arg) 2024-01-13T08:23:51.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:23:51.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:23:51.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:23:51.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: code, '\n'.join(err))) 2024-01-13T08:23:51.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:23:51.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpm3lbpqlt:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpl_srskaa:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:23:51.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:23:51.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:23:51.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:23:51.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 11f5f35c-81e4-4555-9a25-a75daadf41cd 2024-01-13T08:23:51.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:23:51.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:23:51.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:23:51.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:23:51.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:23:51.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr self.prepare() 2024-01-13T08:23:51.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:23:51.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:23:51.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:23:51.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:23:51.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:23:51.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:23:51.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:23:51.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:23:51.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:23:51.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:23:51.652 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:23:51.652 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:23:51.652 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:23:51.652 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:23:51.652 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:23:51.652 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:23:51.653 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:23:51.653 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:23:51.653 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:23:51.653 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:23:51.653 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:23:51.653 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:23:51.653 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:23:51.653 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:23:51.653 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:23:51.653 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:23:51.653 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:23:51.653 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:23:51.653 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:23:51.653 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:23:51.654 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:23:51.654 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:23:51.654 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:23:51.654 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:23:51.654 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:23:51.654 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:23:51.654 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:23:51.654 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:23:51.654 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:23:51.654 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:23:51.654 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:23:51.654 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:23:51.654 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:23:51.654 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:23:51.655 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:23:51.655 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:23:51.655 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:23:51.655 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:23:51.655 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:23:51.655 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:23:51.655 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:23:51.655 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:23:51.655 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: main() 2024-01-13T08:23:51.655 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:23:51.656 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: r = ctx.func(ctx) 2024-01-13T08:23:51.656 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:23:51.656 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:23:51.656 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:23:51.656 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:23:51.656 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:23:51.656 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:23:51.656 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-8777 2024-01-13T08:23:51.656 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: 4f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:23:51.656 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:23:51.656 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:23:51.656 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:23:51.656 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:23:51.657 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:23:51.657 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpm3lbpqlt:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpl_srskaa:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:23:51.657 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: cluster 2024-01-13T08:23:50.319867+0000 mgr.smithi099.oodult (mgr.14180) 228 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:51.657 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: cephadm 2024-01-13T08:23:50.341961+0000 mgr.smithi099.oodult (mgr.14180) 229 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi131', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:23:51.657 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:23:51.657 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:23:51.657 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: return f(*arg) 2024-01-13T08:23:51.657 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:23:51.657 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:23:51.657 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:23:51.657 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: code, '\n'.join(err))) 2024-01-13T08:23:51.657 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi099/config 2024-01-13T08:23:51.657 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpuxber58i:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp847v5n03:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:23:51.657 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:23:51.658 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:23:51.658 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:23:51.658 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 2f9c1a26-e2e9-4648-be87-29b5ac68c198 2024-01-13T08:23:51.658 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:23:51.658 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 --yes-i-really-mean-it 2024-01-13T08:23:51.658 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr stderr: purged osd.1 2024-01-13T08:23:51.658 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:23:51.658 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:23:51.658 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr self.prepare() 2024-01-13T08:23:51.659 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:23:51.659 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:23:51.659 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:23:51.659 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:23:51.659 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:23:51.659 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:23:51.659 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:23:51.659 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:23:51.659 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:23:51.659 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:23:51.659 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:23:51.659 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:23:51.659 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:23:51.659 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:23:51.659 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:23:51.659 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:23:51.660 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:23:51.660 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:23:51.660 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:23:51.660 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:23:51.660 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:23:51.660 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:23:51.661 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:23:51.661 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:23:51.662 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:23:51.662 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:23:51.662 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:23:51.662 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:23:51.662 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:23:51.662 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:23:51.662 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:23:51.662 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:23:51.662 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:23:51.662 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:23:51.662 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:23:51.663 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:23:51.663 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:23:51.663 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:23:51.663 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr rol 2024-01-13T08:23:51.663 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: lback_osd(self.args, self.osd_id) 2024-01-13T08:23:51.663 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:23:51.663 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:23:51.663 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:23:51.663 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:23:51.664 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:23:51.664 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:23:51.664 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:23:51.664 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:23:51.664 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:23:51.664 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:23:51.664 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 1 2024-01-13T08:23:51.664 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:23:51.664 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:23:51.664 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: main() 2024-01-13T08:23:51.664 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:23:51.664 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: r = ctx.func(ctx) 2024-01-13T08:23:51.664 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:23:51.664 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:23:51.665 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:23:51.665 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:23:51.665 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:23:51.665 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:23:51.665 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:23:51.665 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:23:51.665 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:23:51.665 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:23:51.665 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:23:51.665 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:23:51.665 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpuxber58i:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp847v5n03:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:23:51.665 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: cephadm 2024-01-13T08:23:50.343229+0000 mgr.smithi099.oodult (mgr.14180) 230 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T08:23:51.665 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: service_id: all-available-devices 2024-01-13T08:23:51.665 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: service_name: osd.all-available-devices 2024-01-13T08:23:51.666 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: placement: 2024-01-13T08:23:51.666 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: host_pattern: '*' 2024-01-13T08:23:51.666 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: spec: 2024-01-13T08:23:51.666 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: data_devices: 2024-01-13T08:23:51.666 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: all: true 2024-01-13T08:23:51.666 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: filter_logic: AND 2024-01-13T08:23:51.666 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: objectstore: bluestore 2024-01-13T08:23:51.666 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:23:51.667 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpm3lbpqlt:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpl_srskaa:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:23:51.667 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:23:51.667 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:23:51.667 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:23:51.667 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 11f5f35c-81e4-4555-9a25-a75daadf41cd 2024-01-13T08:23:51.667 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:23:51.667 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:23:51.667 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:23:51.667 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:23:51.667 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:23:51.667 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr self.prepare() 2024-01-13T08:23:51.667 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:23:51.668 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:23:51.668 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:23:51.668 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:23:51.668 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:23:51.668 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:23:51.668 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:23:51.668 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:23:51.668 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:23:51.668 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:23:51.668 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:23:51.668 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:23:51.668 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:23:51.669 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:23:51.669 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:23:51.669 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:23:51.669 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:23:51.669 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:23:51.669 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:23:51.669 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:23:51.669 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:23:51.669 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", 2024-01-13T08:23:51.669 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: line 194, in dispatch 2024-01-13T08:23:51.669 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:23:51.669 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:23:51.670 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:23:51.670 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:23:51.670 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:23:51.670 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:23:51.670 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:23:51.670 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:23:51.670 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:23:51.670 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:23:51.670 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:23:51.670 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:23:51.670 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:23:51.670 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:23:51.670 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:23:51.670 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:23:51.671 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:23:51.671 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:23:51.671 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:23:51.671 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:23:51.671 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:23:51.671 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:23:51.671 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:23:51.671 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:23:51.671 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:23:51.671 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:23:51.671 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:23:51.671 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:23:51.671 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:23:51.671 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:23:51.671 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: main() 2024-01-13T08:23:51.672 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:23:51.672 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: r = ctx.func(ctx) 2024-01-13T08:23:51.672 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:23:51.672 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:23:51.672 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:23:51.672 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:23:51.672 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:23:51.672 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:23:51.672 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:23:51.672 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:23:51.672 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:23:51.672 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:23:51.672 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:23:51.673 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:23:51.673 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpm3lbpqlt:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpl_srskaa:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:23:51.673 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:23:51.673 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T08:23:51.673 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: if self._apply_service(spec): 2024-01-13T08:23:51.673 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T08:23:51.673 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T08:23:51.673 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T08:23:51.673 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T08:23:51.674 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T08:23:51.674 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T08:23:51.674 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T08:23:51.674 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T08:23:51.674 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T08:23:51.674 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: raise self._value 2024-01-13T08:23:51.674 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T08:23:51.674 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: result = (True, func(*args, **kwds)) 2024-01-13T08:23:51.674 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T08:23:51.674 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: return list(map(*args)) 2024-01-13T08:23:51.674 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:23:51.674 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: return f(*arg) 2024-01-13T08:23:51.675 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:23:51.675 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:23:51.675 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:23:51.675 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: code, '\n'.join(err))) 2024-01-13T08:23:51.675 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:23:51.675 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpm3lbpqlt:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpl_srskaa:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:23:51.675 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:23:51.675 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:23:51.675 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:23:51.675 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name clie 2024-01-13T08:23:51.675 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: nt.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 11f5f35c-81e4-4555-9a25-a75daadf41cd 2024-01-13T08:23:51.675 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:23:51.676 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:23:51.676 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:23:51.676 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:23:51.676 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:23:51.676 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr self.prepare() 2024-01-13T08:23:51.676 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:23:51.676 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:23:51.676 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:23:51.676 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:23:51.676 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:23:51.676 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:23:51.676 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:23:51.676 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:23:51.677 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:23:51.677 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:23:51.677 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:23:51.677 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:23:51.677 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:23:51.677 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:23:51.677 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:23:51.677 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:23:51.677 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:23:51.678 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:23:51.678 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:23:51.678 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:23:51.678 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:23:51.678 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:23:51.678 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:23:51.678 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:23:51.678 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:23:51.678 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:23:51.678 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:23:51.678 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:23:51.678 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:23:51.678 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:23:51.678 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:23:51.678 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:23:51.679 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:23:51.679 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:23:51.679 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:23:51.679 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:23:51.679 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:23:51.679 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:23:51.679 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:23:51.679 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:23:51.679 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:23:51.679 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:23:51.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:23:51.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:23:51.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:23:51.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:23:51.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:23:51.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:23:51.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:23:51.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:23:51.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:23:51.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:23:51.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: main() 2024-01-13T08:23:51.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:23:51.680 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: r = ctx.func(ctx) 2024-01-13T08:23:51.681 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:23:51.681 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:23:51.681 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:23:51.681 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:23:51.681 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:23:51.681 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:23:51.681 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:23:51.681 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:23:51.681 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:23:51.681 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:23:51.681 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:23:51.681 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:23:51.681 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpm3lbpqlt:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpl_srskaa:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:23:51.681 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:51 smithi131 conmon[38024]: cluster 2024-01-13T08:23:50.343819+0000 mgr.smithi099.oodult (mgr.14180) 231 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:52.386 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:52 smithi099 conmon[33313]: cluster 2024-01-13T08:23:51.266677+0000 mon.smithi099 (mon.0) 584 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T08:23:52.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:52 smithi131 conmon[38024]: cluster 2024-01-13T08:23:51.266677+0000 mon.smithi099 (mon.0) 584 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T08:23:52.694 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:23:53.271 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:23:53.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:53 smithi131 conmon[38024]: cluster 2024-01-13T08:23:52.344290+0000 mgr.smithi099.oodult (mgr.14180) 232 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:53.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:53 smithi131 conmon[38024]: audit 2024-01-13T08:23:52.698431+0000 mon.smithi099 (mon.0) 585 : audit [DBG] from='client.? 172.21.15.99:0/1516443668' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:23:53.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:53 smithi099 conmon[33313]: cluster 2024-01-13T08:23:52.344290+0000 mgr.smithi099.oodult (mgr.14180) 232 : 2024-01-13T08:23:53.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:53 smithi099 conmon[33313]: cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:53.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:53 smithi099 conmon[33313]: audit 2024-01-13T08:23:52.698431+0000 mon.smithi099 (mon.0) 585 : audit [DBG] from='client.? 172.21.15.99:0/1516443668' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:23:54.273 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:23:55.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:55 smithi131 conmon[38024]: cluster 2024-01-13T08:23:54.344778+0000 mgr.smithi099.oodult (mgr.14180) 233 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:55.681 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:55 smithi099 conmon[33313]: cluster 2024-01-13T08:23:54.344778+0000 mgr.smithi099.oodult (mgr.14180) 233 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:55.996 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:23:56.520 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:23:56.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:56 smithi099 conmon[33313]: audit 2024-01-13T08:23:55.999741+0000 mon.smithi099 (mon.0) 586 : audit [DBG] from='client.? 172.21.15.99:0/2706564406' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:23:56.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:56 smithi131 conmon[38024]: audit 2024-01-13T08:23:55.999741+0000 mon.smithi099 (mon.0) 586 : audit [DBG] from='client.? 172.21.15.99:0/2706564406' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:23:57.521 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:23:57.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:57 smithi099 conmon[33313]: cluster 2024-01-13T08:23:56.345445+0000 mgr.smithi099.oodult (mgr.14180) 234 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:57.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:57 smithi131 conmon[38024]: cluster 2024-01-13T08:23:56.345445+0000 mgr.smithi099.oodult (mgr.14180) 234 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:59.286 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:23:59.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:59 smithi099 conmon[33313]: cluster 2024-01-13T08:23:58.346004+0000 mgr.smithi099.oodult (mgr.14180) 235 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:59.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:23:59 smithi099 conmon[33313]: audit 2024-01-13T08:23:59.290831+0000 mon.smithi099 (mon.0) 587 : audit [DBG] from='client.? 172.21.15.99:0/638126331' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:23:59.811 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:23:59.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:59 smithi131 conmon[38024]: cluster 2024-01-13T08:23:58.346004+0000 mgr.smithi099.oodult (mgr.14180) 235 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:23:59.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:59 smithi131 conmon[38024]: audit 2024-01-13T08:23:59.290831+0000 2024-01-13T08:23:59.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:23:59 smithi131 conmon[38024]: mon.smithi099 (mon.0) 587 : audit [DBG] from='client.? 172.21.15.99:0/638126331' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:24:00.812 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:24:01.562 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:01 smithi099 conmon[33313]: cluster 2024-01-13T08:24:00.346418+0000 mgr.smithi099.oodult (mgr 2024-01-13T08:24:01.562 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:01 smithi099 conmon[33313]: .14180) 236 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:01.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:01 smithi131 conmon[38024]: cluster 2024-01-13T08:24:00.346418+0000 mgr.smithi099.oodult (mgr.14180) 236 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:02.504 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:24:03.011 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:24:03.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:03 smithi099 conmon[33313]: cluster 2024-01-13T08:24:02.346798+0000 mgr.smithi099.oodult (mgr.14180) 237 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:03.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:03 smithi099 conmon[33313]: audit 2024-01-13T08:24:02.507668+0000 mon.smithi099 (mon.0) 588 : audit [DBG] from='client.? 172.21.15.99:0/2387600000' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:24:03.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:03 smithi131 conmon[38024]: cluster 2024-01-13T08:24:02.346798+0000 mgr.smithi099.oodult (mgr.14180) 237 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:03.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:03 smithi131 conmon[38024]: audit 2024-01 2024-01-13T08:24:03.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:03 smithi131 conmon[38024]: -13T08:24:02.507668+0000 mon.smithi099 (mon.0) 588 : audit [DBG] from='client.? 172.21.15.99:0/2387600000' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:24:04.012 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:24:05.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:05 smithi099 conmon[33313]: cluster 2024-01-13T08:24:04.347197+0000 mgr.smithi099.oodult (mgr.14180) 238 : 2024-01-13T08:24:05.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:05 smithi099 conmon[33313]: cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:05.742 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:24:05.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:05 smithi131 conmon[38024]: cluster 2024-01-13T08:24:04.347197+0000 mgr.smithi099.oodult (mgr.14180) 238 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:06.278 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:24:06.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:06 smithi099 conmon[33313]: audit 2024-01-13T08:24:05.747082+0000 mon.smithi099 (mon.0) 589 : audit [DBG] from='client.? 172.21.15.99:0/3879949612' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:24:06.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:06 smithi131 conmon[38024]: audit 2024-01-13T08:24:05.747082+0000 mon.smithi099 (mon.0) 589 : audit [DBG] from='client.? 172.21.15.99:0/3879949612' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:24:07.279 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:24:07.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:07 smithi099 conmon[33313]: cluster 2024-01-13T08:24:06.347596+0000 mgr.smithi099.oodult (mgr.14180) 239 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:07.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:07 smithi131 conmon[38024]: cluster 2024-01-13T08:24:06.347596+0000 mgr.smithi099.oodult (mgr.14180) 239 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:09.070 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:24:09.617 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:24:09.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:09 smithi099 conmon[33313]: cluster 2024-01-13T08:24:08.348103+0000 mgr.smithi099.oodult (mgr.14180) 240 : cluster 2024-01-13T08:24:09.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:09 smithi099 conmon[33313]: [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:09.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:09 smithi099 conmon[33313]: audit 2024-01-13T08:24:09.074915+0000 mon.smithi099 (mon 2024-01-13T08:24:09.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:09 smithi099 conmon[33313]: .0) 590 : audit [DBG] from='client.? 172.21.15.99:0/1792963303' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:24:09.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:09 smithi131 conmon[38024]: cluster 2024-01-13T08:24:08.348103+0000 mgr.smithi099.oodult (mgr.14180) 240 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:09.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:09 smithi131 conmon[38024]: audit 2024-01-13T08:24:09.074915+0000 mon.smithi099 2024-01-13T08:24:09.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:09 smithi131 conmon[38024]: (mon.0) 590 : audit [DBG] from='client.? 172.21.15.99:0/1792963303' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:24:10.617 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:24:11.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:11 smithi099 conmon[33313]: cluster 2024-01-13T08:24:10.348667+0000 mgr.smithi099.oodult (mgr.14180) 241 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:11.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:11 smithi131 conmon[38024]: cluster 2024-01-13T08:24:10.348667+0000 mgr.smithi099.oodult (mgr.14180) 241 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:12.339 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:24:12.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:12 smithi099 conmon[33313]: audit 2024-01-13T08:24:12.342788+0000 mon.smithi099 (mon.0) 591 2024-01-13T08:24:12.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:12 smithi099 conmon[33313]: : audit [DBG] from='client.? 172.21.15.99:0/1653355448' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:24:12.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:12 smithi131 conmon[38024]: audit 2024-01-13T08:24:12.342788+0000 mon.smithi099 (mon.0) 591 : audit [DBG] from='client.? 172.21.15.99:0/1653355448' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:24:12.909 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:24:13.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:13 smithi099 conmon[33313]: cluster 2024-01-13T08:24:12.349055+0000 mgr.smithi099.oodult (mgr.14180) 242 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:13.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:13 smithi131 conmon[38024]: cluster 2024-01-13T08:24:12.349055+0000 mgr.smithi099.oodult (mgr.14180) 242 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:13.910 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:24:15.676 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:24:15.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:15 smithi099 conmon[33313]: cluster 2024-01-13T08:24:14.349586+0000 mgr.smithi099.oodult (mgr.14180) 243 2024-01-13T08:24:15.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:15 smithi099 conmon[33313]: : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:15.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:15 smithi131 conmon[38024]: cluster 2024-01-13T08:24:14.349586+0000 mgr.smithi099.oodult (mgr.14180) 243 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:16.241 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:24:16.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:16 smithi099 conmon[33313]: audit 2024-01-13T08:24:15.679974+0000 mon.smithi099 (mon.0) 592 : audit [DBG] from='client.? 172.21.15.99:0/153616343' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:24:16.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:16 smithi131 conmon[38024]: audit 2024-01-13T08:24:15.679974+0000 mon.smithi099 (mon.0) 592 : audit [DBG] from='client.? 172.21.15.99:0/153616343' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:24:17.242 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:24:17.682 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:17 smithi099 conmon[33313]: cluster 2024-01-13T08:24:16.349976+0000 mgr.smithi099.oodult (mgr. 2024-01-13T08:24:17.682 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:17 smithi099 conmon[33313]: 14180) 244 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:17.898 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:17 smithi131 conmon[38024]: cluster 2024-01-13T08:24:16.349976+0000 mgr.smithi099.oodult (mgr.14180) 244 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:19.059 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:24:19.564 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:24:19.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:19 smithi099 conmon[33313]: cluster 2024-01-13T08:24:18.350348+0000 mgr.smithi099.oodult (mgr.14180) 245 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:19.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:19 smithi099 conmon[33313]: audit 2024-01-13T08:24:19.062979+0000 mon.smithi099 (mon.0) 593 : audit [DBG] from='client.? 172.21.15.99:0/2087852106' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:24:19.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:19 smithi131 conmon[38024]: cluster 2024-01-13T08:24:18.350348+0000 mgr.smithi099.oodult (mgr.14180) 245 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:19.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:19 smithi131 conmon[38024]: audit 2024-01-13T08:24:19.062979+0000 mon.smithi099 (mon 2024-01-13T08:24:19.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:19 smithi131 conmon[38024]: .0) 593 : audit [DBG] from='client.? 172.21.15.99:0/2087852106' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:24:20.565 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:24:21.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:21 smithi099 conmon[33313]: cluster 2024-01-13T08:24:20. 2024-01-13T08:24:21.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:21 smithi099 conmon[33313]: 350755+0000 mgr.smithi099.oodult (mgr.14180) 246 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:21.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:21 smithi131 conmon[38024]: cluster 2024-01-13T08:24:20.350755+0000 mgr.smithi099.oodult (mgr.14180) 246 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:22.336 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:24:22.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:22 smithi099 conmon[33313]: audit 2024-01-13T08:24:22.340618+0000 mon.smithi099 (mon.0) 594 : audit [DBG] from='client.? 172.21.15.99:0/597665530' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:24:22.838 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:24:22.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:22 smithi131 conmon[38024]: audit 2024-01-13T08:24:22.340618+0000 mon.smithi099 (mon.0) 594 : audit [DBG] from='client.? 172.21.15.99:0/597665530' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:24:23.839 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:24:23.873 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:23 smithi099 conmon[33313]: cluster 2024-01-13T08:24:22.351127+0000 mgr.smithi099.oodult (mgr.14180) 247 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:23.898 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:23 smithi131 conmon[38024]: cluster 2024-01-13T08:24:22.351127+0000 mgr.smithi099.oodult (mgr.14180) 247 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:25.603 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:24:25.633 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:25 smithi099 conmon[33313]: cluster 2024-01-13T08:24:24.351598+0000 mgr.smithi099.oodult (mgr.14180) 248 : cluster 2024-01-13T08:24:25.633 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:25 smithi099 conmon[33313]: [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:25.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:25 smithi131 conmon[38024]: cluster 2024-01-13T08:24:24.351598+0000 mgr.smithi099.oodult (mgr.14180) 248 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:26.165 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:24:26.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:26 smithi131 conmon[38024]: audit 2024-01-13T08:24:25.606836+0000 mon.smithi099 (mon.0) 595 : audit [DBG] from='client.? 172.21.15.99:0/777424739' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:24:26.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:26 smithi099 conmon[33313]: audit 2024-01-13T08:24:25.606836+0000 mon.smithi099 (mon.0) 595 : audit [DBG] from='client.? 172.21.15.99:0/777424739' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:24:27.166 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:24:27.790 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:27 smithi099 conmon[33313]: cluster 2024-01-13T08:24:26.352030 2024-01-13T08:24:27.791 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:27 smithi099 conmon[33313]: +0000 mgr.smithi099.oodult (mgr.14180) 249 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:27.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:27 smithi131 conmon[38024]: cluster 2024-01-13T08:24:26.352030+0000 mgr.smithi099.oodult (mgr.14180) 249 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:28.872 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:24:29.471 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:24:29.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:29 smithi099 conmon[33313]: cluster 2024-01-13T08:24:28.352451+0000 mgr.smithi099.oodult (mgr.14180) 2024-01-13T08:24:29.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:29 smithi099 conmon[33313]: 250 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:29.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:29 smithi099 conmon[33313]: audit 2024-01-13T08:24:28.876705+0000 mon.smithi099 (mon.0) 596 : audit [DBG] from='client.? 172.21.15.99:0/1728155218' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:24:29.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:29 smithi099 conmon[33313]: 2024-01-13T08:24:29.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:29 smithi131 conmon[38024]: cluster 2024-01-13T08:24:28.352451+0000 mgr.smithi099.oodult (mgr.14180) 250 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:29.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:29 smithi131 conmon[38024]: audit 2024-01-13T08:24:28.876705+0000 mon.smithi099 (mon.0) 596 : audit [DBG] from='client.? 172.21.15.99:0/1728155218' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:24:30.473 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:24:31.847 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:31 smithi099 conmon[33313]: cluster 2024-01-13T08:24:30.352922+0000 mgr.smithi099.oodult (mgr.14180) 251 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:31.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:31 smithi131 conmon[38024]: cluster 2024-01-13T08:24:30.352922+0000 mgr.smithi099.oodult (mgr.14180) 251 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:32.197 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:24:32.769 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:32 smithi099 conmon[33313]: audit 2024-01-13T08:24:32.201123+0000 mon.smithi099 (mon.0) 597 : audit [DBG] 2024-01-13T08:24:32.769 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:32 smithi099 conmon[33313]: from='client.? 172.21.15.99:0/717527333' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:24:32.770 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:24:32.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:32 smithi131 conmon[38024]: audit 2024-01-13T08:24:32.201123+0000 mon.smithi099 (mon.0) 597 : audit [DBG] from='client.? 172.21.15.99:0/717527333' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:24:33.771 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:24:33.805 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:33 smithi099 conmon[33313]: cluster 2024-01-13T08:24:32.353484+0000 mgr.smithi099.oodult (mgr.14180) 252 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:33.898 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:33 smithi131 conmon[38024]: cluster 2024-01-13T08:24:32.353484+0000 mgr.smithi099.oodult (mgr.14180) 252 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:35.560 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:24:35.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:35 smithi131 conmon[38024]: cluster 2024-01-13T08:24:34.354189+0000 mgr.smithi099.oodult (mgr.14180) 253 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:35.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:35 smithi099 conmon[33313]: cluster 2024-01-13T08:24:34.354189+0000 mgr.smithi099.oodult (mgr.14180) 253 : cluster 2024-01-13T08:24:35.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:35 smithi099 conmon[33313]: [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:36.136 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:24:36.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:36 smithi131 conmon[38024]: audit 2024-01-13T08:24:35.564204+0000 mon.smithi099 (mon.0) 598 : audit [DBG] from='client.? 172.21.15.99:0/345188155' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:24:36.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:36 smithi099 conmon[33313]: audit 2024-01-13T08:24:35.564204+0000 mon.smithi099 (mon.0) 2024-01-13T08:24:36.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:36 smithi099 conmon[33313]: 598 : audit [DBG] from='client.? 172.21.15.99:0/345188155' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:24:37.138 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:24:37.898 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:37 smithi131 conmon[38024]: cluster 2024-01-13T08:24:36.354816+0000 mgr.smithi099.oodult (mgr.14180) 254 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:37.945 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:37 smithi099 conmon[33313]: cluster 2024-01-13T08:24:36.354816+0000 mgr.smithi099.oodult (mgr.14180) 254 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:38.965 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:24:39.485 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:24:39.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:39 smithi099 conmon[33313]: cluster 2024-01-13T08:24:38.355230+0000 mgr.smithi099.oodult (mgr.14180) 255 2024-01-13T08:24:39.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:39 smithi099 conmon[33313]: : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:39.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:39 smithi099 conmon[33313]: audit 2024-01-13T08:24:38.969696+0000 mon.smithi099 (mon.0) 599 : audit [DBG] from='client.? 172.21.15.99:0/3779083182' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:24:39.898 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:39 smithi131 conmon[38024]: cluster 2024-01-13T08:24:38.355230+0000 mgr.smithi099.oodult (mgr.14180) 255 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:39.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:39 smithi131 conmon[38024]: audit 2024-01-13T08:24:38.969696+0000 mon.smithi099 (mon.0) 599 : 2024-01-13T08:24:39.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:39 smithi131 conmon[38024]: audit [DBG] from='client.? 172.21.15.99:0/3779083182' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:24:40.487 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:24:41.862 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:41 smithi099 conmon[33313]: cluster 2024-01-13T08:24:40.355919+0000 mgr.smithi099.oodult (mgr.14180) 256 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:41.898 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:41 smithi131 conmon[38024]: cluster 2024-01-13T08:24:40.355919+0000 mgr.smithi099.oodult (mgr.14180) 256 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:42.212 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:24:42.751 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:24:42.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:42 smithi131 conmon[38024]: audit 2024-01-13T08:24:42.217046+0000 mon.smithi099 (mon.0) 600 : audit [DBG] from='client.? 172.21.15.99:0/1414709196' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:24:42.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:42 smithi099 conmon[33313]: audit 2024-01-13T08:24:42.217046+0000 mon.smithi099 (mon.0) 600 : audit [DBG] 2024-01-13T08:24:42.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:42 smithi099 conmon[33313]: from='client.? 172.21.15.99:0/1414709196' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:24:43.753 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:24:43.898 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:43 smithi131 conmon[38024]: cluster 2024-01-13T08:24:42.356440+0000 mgr.smithi099.oodult (mgr.14180) 257 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:43.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:43 smithi099 conmon[33313]: cluster 2024-01-13T08:24:42.356440+0000 mgr.smithi099.oodult (mgr.14180) 257 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:45.492 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:24:45.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:45 smithi131 conmon[38024]: cluster 2024-01-13T08:24:44.356966+0000 mgr.smithi099.oodult (mgr.14180) 258 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:45.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:45 smithi131 conmon[38024]: audit 2024-01-13T08:24:45.496322+0000 mon.smithi099 (mon. 2024-01-13T08:24:45.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:45 smithi131 conmon[38024]: 0) 601 : audit [DBG] from='client.? 172.21.15.99:0/1424723950' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:24:45.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:45 smithi099 conmon[33313]: cluster 2024-01-13T08:24:44.356966+0000 mgr.smithi099.oodult (mgr.14180) 258 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:45.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:45 smithi099 conmon[33313]: audit 2024-01-13T08:24:45.496322+0000 2024-01-13T08:24:45.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:45 smithi099 conmon[33313]: mon.smithi099 (mon.0) 601 : audit [DBG] from='client.? 172.21.15.99:0/1424723950' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:24:46.033 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:24:47.034 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:24:47.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:47 smithi131 conmon[38024]: cluster 2024-01-13T08:24:46.357493+0000 mgr.smithi099.oodult (mgr.14180) 259 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:47.475 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:47 smithi099 conmon[33313]: cluster 2024-01-13T08:24:46.357493+0000 mgr.smithi099.oodult (mgr.14180) 259 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:48.762 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:24:49.342 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:24:49.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:49 smithi099 conmon[33313]: cluster 2024-01-13T08:24:48.357963+0000 mgr.smithi099.oodult (mgr.14180) 260 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:49.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:49 smithi099 conmon[33313]: audit 2024-01-13T08:24: 2024-01-13T08:24:49.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:49 smithi099 conmon[33313]: 48.766328+0000 mon.smithi099 (mon.0) 602 : audit [DBG] from='client.? 172.21.15.99:0/3091696126' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:24:49.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:49 smithi131 conmon[38024]: cluster 2024-01-13T08:24:48.357963+0000 mgr.smithi099.oodult (mgr.14180) 260 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:49.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:49 smithi131 conmon[38024]: audit 2024-01-13T08:24:48.766328+0000 mon.smithi099 (mon.0) 602 : audit [DBG] 2024-01-13T08:24:49.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:49 smithi131 conmon[38024]: from='client.? 172.21.15.99:0/3091696126' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:24:50.343 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:24:50.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:50 smithi099 conmon[33313]: audit 2024-01-13T08:24:50.344886+0000 mon.smithi099 (mon.0) 603 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:24:50.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:50 smithi131 conmon[38024]: audit 2024-01-13T08:24:50.344886+0000 mon.smithi099 (mon.0) 603 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-13T08:24:51.569 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:51 smithi131 conmon[38024]: cluster 2024-01-13T08:24:50.358403+0000 mgr.smithi099.oodult (mgr.14180) 261 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:51.569 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:51 smithi131 conmon[38024]: audit 2024-01-13T08:24:50.693984+0000 mon.smithi099 (mon.0) 604 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:24:51.569 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:51 smithi131 conmon[38024]: audit 2024-01-13T08:24:50.694634+0000 mon.smithi099 (mon.0) 605 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T08:24:51.569 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:51 smithi131 conmon[38024]: audit 2024-01-13T08:24:50.698910+0000 mon.smithi099 (mon.0) 606 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:24:51.569 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:51 smithi131 conmon[38024]: cluster 2024-01-13T08:24:50.699229+0000 mgr.smithi099.oodult (mgr.14180) 262 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:51.569 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:51 smithi131 conmon[38024]: audit 2024-01-13T08:24:50.702188+0000 mon.smithi099 (mon.0) 607 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T08:24:51.569 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:51 smithi131 conmon[38024]: audit 2024-01-13T08:24:50.704822+0000 mon.smithi099 (mon.0) 608 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:24:51.569 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:51 smithi131 conmon[38024]: audit 2024-01-13T08: 2024-01-13T08:24:51.569 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:51 smithi131 conmon[38024]: 24:50.705682+0000 mon.smithi099 (mon.0) 609 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:24:51.569 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:51 smithi131 conmon[38024]: audit 2024-01-13T08:24:50.705848+0000 mon.smithi099 (mon.0) 610 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:24:51.569 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:51 smithi131 conmon[38024]: audit 2024-01-13T08:24:50.706294+0000 mon.smithi099 (mon.0) 611 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:24:51.701 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:51 smithi099 conmon[33313]: cluster 2024-01-13T08:24:50.358403+0000 mgr.smithi099.oodult (mgr.14180) 261 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:51.701 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:51 smithi099 conmon[33313]: audit 2024-01-13T08:24:50.693984+0000 mon.smithi099 (mon.0) 604 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:24:51.701 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:51 smithi099 conmon[33313]: audit 2024-01-13T08:24:50.694634+0000 mon.smithi099 (mon.0) 605 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-13T08:24:51.701 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:51 smithi099 conmon[33313]: audit 2024-01-13T08:24:50.698910+0000 mon.smithi099 (mon.0) 606 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:24:51.701 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:51 smithi099 conmon[33313]: cluster 2024-01-13T08:24:50.699229+0000 mgr.smithi099.oodult (mgr.14180) 262 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:51.701 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:51 smithi099 conmon[33313]: audit 2024-01-13T08:24:50.702188+0000 mon.smithi099 (mon.0) 607 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-13T08:24:51.702 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:51 smithi099 conmon[33313]: audit 2024-01-13T08:24:50.704822+0000 mon.smithi099 (mon.0) 608 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:24:51.702 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:51 smithi099 conmon[33313]: audit 2024-01-13T08:24:50.705682+0000 mon.smithi099 (mon.0) 609 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-13T08:24:51.702 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:51 smithi099 conmon[33313]: audit 2024-01-13T08:24:50.705848+0000 mon.smithi099 (mon.0) 610 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:24:51.702 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:51 smithi099 conmon[33313]: audit 2024-01-13T08:24:50.706294+0000 mon.smithi099 (mon.0) 611 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-13T08:24:52.106 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:24:52.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:52 smithi131 conmon[38024]: cluster 2024-01-13T08:24:51.697184+0000 mon.smithi099 (mon.0) 612 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T08:24:52.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:52 smithi131 conmon[38024]: cluster 2024-01-13T08:24:51.697209+0000 mon.smithi099 (mon.0) 613 : cluster [INF] Cluster is now healthy 2024-01-13T08:24:52.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:52 smithi131 conmon[38024]: audit 2024-01-13T08:24:52.079080+0000 mon.smithi099 (mon.0) 614 : audit [DBG] from='client.? 172.21.15.99:0/4190156491' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:24:52.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:52 smithi131 conmon[38024]: audit 2024-01-13T08:24:52.250531+0000 mon.smithi099 (mon.0) 615 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:24:52.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:52 smithi099 conmon[33313]: cluster 2024-01-13T08:24:51.697184+0000 mon.smithi099 (mon.0) 612 : cluster [INF] Health check cleared: CEPHADM_APPLY_SPEC_FAIL (was: Failed to apply 1 service(s): osd.all-available-devices) 2024-01-13T08:24:52.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:52 smithi099 conmon[33313]: cluster 2024-01-13T08:24:51.697209+0000 mon.smithi099 (mon.0) 613 : cluster [INF] Cluster is now healthy 2024-01-13T08:24:52.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:52 smithi099 conmon[33313]: audit 2024-01-13T08:24:52.079080+0000 mon.smithi099 (mon.0) 614 : audit [DBG] from='client.? 172.21.15.99:0/4190156491' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:24:52.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:52 smithi099 conmon[33313]: audit 2024-01-13T08:24:52.250531+0000 mon.smithi099 (mon.0) 615 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:24:53.056 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":29,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:24:53.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:53 smithi099 conmon[33313]: cluster 2024-01-13T08:24:52.699537+0000 mgr.smithi099.oodult (mgr.14180) 263 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:53.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:53 smithi099 conmon[33313]: audit 2024-01-13T08:24:53.258364+0000 mon.smithi131 (mon.1) 14 : audit [INF] from='client.? 172.21.15.131:0/1621045429' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b27664f-50ee-481c-845c-8da9454e9617"}]: dispatch 2024-01-13T08:24:53.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:53 smithi099 conmon[33313]: audit 2024-01-13T08:24:53.258759+0000 mon.smithi099 (mon.0) 616 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b27664f-50ee-481c-845c-8da9454e9617"}]: dispatch 2024-01-13T08:24:53.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:53 smithi099 conmon[33313]: audit 2024-01-13T08:24:53.262734+0000 mon.smithi099 (mon.0) 617 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1b27664f-50ee-481c-845c-8da9454e9617"}]': finished 2024-01-13T08:24:53.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:53 smithi099 conmon[33313]: cluster 2024-01-13T08:24:53.262778+0000 mon.smithi099 (mon.0) 618 : cluster [DBG] osdmap e30: 1 total, 0 up, 1 in 2024-01-13T08:24:53.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:53 smithi099 conmon[33313]: audit 2024-01-13T08:24:53.262853+0000 mon.smithi099 (mon.0) 619 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:24:53.818 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:53 smithi131 conmon[38024]: cluster 2024-01-13T08:24:52.699537+0000 mgr.smithi099.oodult (mgr.14180) 263 : 2024-01-13T08:24:53.818 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:53 smithi131 conmon[38024]: cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:53.818 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:53 smithi131 conmon[38024]: audit 2024-01-13T08:24:53.258364+0000 mon.smithi131 (mon.1) 14 : audit [INF] from='client.? 172.21.15.131:0/1621045429' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b27664f-50ee-481c-845c-8da9454e9617"}]: dispatch 2024-01-13T08:24:53.819 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:53 smithi131 conmon[38024]: audit 2024-01-13T08:24:53.258759+0000 mon.smithi099 (mon.0) 616 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b27664f-50ee-481c-845c-8da9454e9617"}]: dispatch 2024-01-13T08:24:53.819 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:53 smithi131 conmon[38024]: audit 2024-01-13T08:24:53.262734+0000 mon.smithi099 (mon.0) 617 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1b27664f-50ee-481c-845c-8da9454e9617"}]': finished 2024-01-13T08:24:53.819 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:53 smithi131 conmon[38024]: cluster 2024-01-13T08:24:53.262778+0000 mon.smithi099 (mon.0) 618 : cluster [DBG] osdmap e30: 1 total, 0 up, 1 in 2024-01-13T08:24:53.819 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:53 smithi131 conmon[38024]: audit 2024-01-13T08:24:53.262853+0000 mon.smithi099 (mon.0) 619 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:24:54.057 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:24:54.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:54 smithi131 conmon[38024]: audit 2024-01-13T08:24:53.552648+0000 mon.smithi131 (mon.1) 15 : audit [INF] 2024-01-13T08:24:54.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:54 smithi131 conmon[38024]: from='client.? 172.21.15.131:0/2565475243' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:24:54.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:54 smithi131 conmon[38024]: audit 2024-01-13T08:24:53.553079+0000 mon.smithi099 (mon.0) 620 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:24:54.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:54 smithi131 conmon[38024]: audit 2024-01-13T08 2024-01-13T08:24:54.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:54 smithi131 conmon[38024]: :24:53.559934+0000 mon.smithi099 (mon.0) 621 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:24:54.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:54 smithi131 conmon[38024]: cluster 2024-01-13T 2024-01-13T08:24:54.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:54 smithi131 conmon[38024]: 08:24:53.559987+0000 mon.smithi099 ( 2024-01-13T08:24:54.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:54 smithi131 conmon[38024]: mon.0) 622 : cluster [DBG] osdmap e31: 0 total, 0 up, 0 in 2024-01-13T08:24:54.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:54 smithi131 conmon[38024]: audit 2024-01-13T08:24:54.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:54 smithi131 conmon[38024]: 2024-01-13T08:24 2024-01-13T08:24:54.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:54 smithi131 conmon[38024]: :54.332029+0000 mon.smithi099 (mon.0) 2024-01-13T08:24:54.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:54 smithi131 conmon[38024]: 623 : audit [INF] from='client.? 172.21.15.99:0/4109954675' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d5544d4-83c1-47b6-9aa7-0f362d68e82c"}]: dispatch 2024-01-13T08:24:54.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:54 smithi131 conmon[38024]: audit 2024-01 2024-01-13T08:24:54.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:54 smithi131 conmon[38024]: -13T08:24:54.338680 2024-01-13T08:24:54.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:54 smithi131 conmon[38024]: +0000 mon.smithi099 (mon.0) 624 : audit [INF] 2024-01-13T08:24:54.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:54 smithi131 conmon[38024]: from='client.? 172.21.15.99:0/4109954675' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3d5544d4-83c1-47b6-9aa7-0f362d68e82c"}]': finished 2024-01-13T08:24:54.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:54 smithi131 conmon[38024]: cluster 2024-01- 2024-01-13T08:24:54.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:54 smithi131 conmon[38024]: 13T08:24:54.338725 2024-01-13T08:24:54.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:54 smithi131 conmon[38024]: +0000 mon.smithi099 (mon.0) 625 : cluster [DBG] 2024-01-13T08:24:54.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:54 smithi131 conmon[38024]: osdmap e32: 1 total, 0 up, 1 in 2024-01-13T08:24:54.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:54 smithi131 conmon[38024]: audit 2024-01-13T08:24:54.650 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:54 smithi131 conmon[38024]: 2024-01-13T08:24 2024-01-13T08:24:54.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:54 smithi131 conmon[38024]: :54.338866+0000 mon.smithi099 (mon.0) 626 2024-01-13T08:24:54.651 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:54 smithi131 conmon[38024]: : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:24:54.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:54 smithi099 conmon[33313]: audit 2024-01-13T08:24:53.552648+0000 mon.smithi131 (mon.1) 15 : audit [INF] from='client.? 172.21.15.131:0/2565475243' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:24:54.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:54 smithi099 conmon[33313]: audit 2024-01-13T08:24:53.553079+0000 mon.smithi099 (mon.0) 620 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:24:54.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:54 smithi099 conmon[33313]: audit 2024-01-13T08:24:53.559934+0000 mon.smithi099 (mon.0) 621 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:24:54.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:54 smithi099 conmon[33313]: cluster 2024-01-13T08:24:53.559987+0000 mon.smithi099 (mon.0) 622 : cluster [DBG] osdmap e31: 0 total, 0 up, 0 in 2024-01-13T08:24:54.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:54 smithi099 conmon[33313]: audit 2024-01-13T08:24:54.332029+0000 mon.smithi099 (mon.0) 623 : audit [INF] from='client.? 172.21.15.99:0/4109954675' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d5544d4-83c1-47b6-9aa7-0f362d68e82c"}]: dispatch 2024-01-13T08:24:54.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:54 smithi099 conmon[33313]: audit 2024-01-13T08:24:54.338680+0000 mon.smithi099 (mon.0) 624 : audit [INF] from='client.? 172.21.15.99:0/4109954675' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3d5544d4-83c1-47b6-9aa7-0f362d68e82c"}]': finished 2024-01-13T08:24:54.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:54 smithi099 conmon[33313]: cluster 2024-01-13T08:24:54.338725+0000 mon.smithi099 (mon.0) 625 : cluster 2024-01-13T08:24:54.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:54 smithi099 conmon[33313]: [DBG] osdmap e32: 1 total, 0 up, 1 in 2024-01-13T08:24:54.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:54 smithi099 conmon[33313]: audit 2024-01-13T08:24:54.338866+0000 mon.smithi099 (mon.0) 626 : audit [DBG] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-13T08:24:55.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: cephadm 2024-01-13T08:24:54.245101+0000 mgr.smithi099.oodult (mgr.14180) 264 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi131', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:24:55.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:24:55.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:24:55.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: return f(*arg) 2024-01-13T08:24:55.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:24:55.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:24:55.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:24:55.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: code, '\n'.join(err))) 2024-01-13T08:24:55.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:24:55.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp5oq8mucz:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbeyo9n8v:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:24:55.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:24:55.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:24:55.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:24:55.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 1b27664f-50ee-481c-845c-8da9454e9617 2024-01-13T08:24:55.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:24:55.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:24:55.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:24:55.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:24:55.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:24:55.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr self.prepare() 2024-01-13T08:24:55.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:24:55.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:24:55.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:24:55.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:24:55.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:24:55.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:24:55.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:24:55.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:24:55.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:24:55.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:24:55.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:24:55.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:24:55.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:24:55.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:24:55.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:24:55.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:24:55.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:24:55.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:24:55.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:24:55.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:24:55.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:24:55.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:24:55.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:24:55.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:24:55.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:24:55.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:24:55.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:24:55.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:24:55.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:24:55.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:24:55.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:24:55.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:24:55.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:24:55.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:24:55.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:24:55.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:24:55.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:24:55.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:24:55.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:24:55.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:24:55.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:24:55.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:24:55.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:24:55.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:24:55.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:24:55.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:24:55.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:24:55.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:24:55.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:24:55.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:24:55.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:24:55.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:24:55.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: main() 2024-01-13T08:24:55.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:24:55.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: r = ctx.func(ctx) 2024-01-13T08:24:55.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:24:55.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:24:55.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:24:55.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:24:55.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:24:55.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:24:55.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-8777 2024-01-13T08:24:55.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: 4f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:24:55.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:24:55.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:24:55.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:24:55.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:24:55.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:24:55.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp5oq8mucz:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbeyo9n8v:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:24:55.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: audit 2024-01-13T08:24:54.651313+0000 mon.smithi099 (mon.0) 627 : audit [INF] from='client.? 172.21.15.99:0/1735874441' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:24:55.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: audit 2024-01-13T08:24:54.664491+0000 mon.smithi099 (mon.0) 628 : audit [INF] from='client.? 172.21.15.99:0/1735874441' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:24:55.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: cluster 2024-01-13T08:24:54.664557+0000 mon.smithi099 (mon.0) 629 : cluster [DBG] osdmap e33: 0 total, 0 up, 0 in 2024-01-13T08:24:55.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:55 smithi099 conmon[33313]: cluster 2024-01-13T08:24:54.699783+0000 mgr.smithi099.oodult (mgr.14180) 265 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:55.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: cephadm 2024-01-13T08:24:54.245101+0000 mgr.smithi099.oodult (mgr.14180) 264 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi131', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:24:55.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:24:55.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:24:55.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: return f(*arg) 2024-01-13T08:24:55.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:24:55.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:24:55.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:24:55.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: code, '\n'.join(err))) 2024-01-13T08:24:55.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:24:55.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp5oq8mucz:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbeyo9n8v:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:24:55.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:24:55.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:24:55.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:24:55.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 1b27664f-50ee-481c-845c-8da9454e9617 2024-01-13T08:24:55.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:24:55.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:24:55.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:24:55.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:24:55.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:24:55.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr self.prepare() 2024-01-13T08:24:55.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:24:55.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:24:55.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:24:55.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:24:55.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:24:55.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:24:55.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:24:55.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:24:55.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:24:55.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:24:55.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:24:55.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:24:55.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:24:55.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:24:55.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:24:55.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:24:55.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:24:55.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:24:55.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:24:55.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:24:55.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:24:55.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:24:55.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:24:55.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:24:55.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:24:55.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:24:55.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:24:55.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:24:55.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:24:55.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:24:55.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:24:55.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:24:55.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:24:55.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:24:55.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:24:55.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:24:55.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:24:55.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:24:55.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:24:55.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:24:55.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:24:55.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:24:55.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:24:55.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:24:55.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:24:55.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:24:55.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:24:55.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:24:55.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:24:55.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:24:55.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:24:55.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:24:55.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: main() 2024-01-13T08:24:55.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:24:55.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: r = ctx.func(ctx) 2024-01-13T08:24:55.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:24:55.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:24:55.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:24:55.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:24:55.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:24:55.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:24:55.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-8777 2024-01-13T08:24:55.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: 4f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:24:55.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:24:55.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:24:55.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:24:55.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:24:55.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:24:55.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp5oq8mucz:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbeyo9n8v:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:24:55.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: audit 2024-01-13T08:24:54.651313+0000 mon.smithi099 (mon.0) 627 : audit [INF] from='client.? 172.21.15.99:0/1735874441' entity='client.bootstrap-osd' cmd=[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]: dispatch 2024-01-13T08:24:55.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: audit 2024-01-13T08:24:54.664491+0000 mon.smithi099 (mon.0) 628 : audit [INF] from='client.? 172.21.15.99:0/1735874441' entity='client.bootstrap-osd' cmd='[{"prefix": "osd purge-new", "id": 0, "yes_i_really_mean_it": true}]': finished 2024-01-13T08:24:55.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: cluster 2024-01-13T08:24:54.664557+0000 mon.smithi099 (mon.0) 629 : cluster [DBG] osdmap e33: 0 total, 0 up, 0 in 2024-01-13T08:24:55.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:55 smithi131 conmon[38024]: cluster 2024-01-13T08:24:54.699783+0000 mgr.smithi099.oodult (mgr.14180) 265 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:56.269 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:24:56.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: cephadm 2024-01-13T08:24:55.605648+0000 mgr.smithi099.oodult (mgr.14180) 266 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi131', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:24:56.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:24:56.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:24:56.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: return f(*arg) 2024-01-13T08:24:56.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:24:56.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:24:56.732 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:24:56.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: code, '\n'.join(err))) 2024-01-13T08:24:56.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi099/config 2024-01-13T08:24:56.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpbo572j8d:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpzrvovat9:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:24:56.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:24:56.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:24:56.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:24:56.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 3d5544d4-83c1-47b6-9aa7-0f362d68e82c 2024-01-13T08:24:56.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:24:56.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:24:56.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:24:56.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:24:56.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:24:56.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr self.prepare() 2024-01-13T08:24:56.733 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:24:56.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:24:56.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:24:56.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:24:56.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:24:56.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:24:56.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:24:56.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:24:56.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:24:56.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:24:56.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:24:56.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:24:56.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:24:56.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:24:56.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:24:56.734 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:24:56.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:24:56.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:24:56.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:24:56.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:24:56.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:24:56.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:24:56.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:24:56.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:24:56.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:24:56.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:24:56.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:24:56.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:24:56.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:24:56.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:24:56.735 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:24:56.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:24:56.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:24:56.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:24:56.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:24:56.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:24:56.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:24:56.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:24:56.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:24:56.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:24:56.736 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:24:56.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:24:56.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:24:56.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:24:56.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:24:56.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:24:56.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:24:56.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:24:56.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:24:56.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:24:56.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:24:56.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:24:56.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: main() 2024-01-13T08:24:56.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:24:56.737 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: r = ctx.func(ctx) 2024-01-13T08:24:56.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:24:56.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:24:56.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:24:56.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:24:56.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:24:56.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:24:56.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-8777 2024-01-13T08:24:56.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: 4f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:24:56.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:24:56.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:24:56.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:24:56.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:24:56.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:24:56.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpbo572j8d:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpzrvovat9:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:24:56.738 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: cephadm 2024-01-13T08:24:55.607566+0000 mgr.smithi099.oodult (mgr.14180) 267 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T08:24:56.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: service_id: all-available-devices 2024-01-13T08:24:56.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: service_name: osd.all-available-devices 2024-01-13T08:24:56.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: placement: 2024-01-13T08:24:56.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: host_pattern: '*' 2024-01-13T08:24:56.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: spec: 2024-01-13T08:24:56.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: data_devices: 2024-01-13T08:24:56.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: all: true 2024-01-13T08:24:56.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: filter_logic: AND 2024-01-13T08:24:56.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: objectstore: bluestore 2024-01-13T08:24:56.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:24:56.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp5oq8mucz:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbeyo9n8v:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:24:56.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:24:56.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:24:56.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:24:56.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 1b27664f-50ee-481c-845c-8da9454e9617 2024-01-13T08:24:56.739 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:24:56.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:24:56.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:24:56.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:24:56.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:24:56.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr self.prepare() 2024-01-13T08:24:56.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:24:56.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:24:56.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:24:56.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:24:56.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:24:56.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:24:56.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:24:56.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:24:56.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:24:56.740 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:24:56.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:24:56.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:24:56.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:24:56.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:24:56.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:24:56.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:24:56.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:24:56.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:24:56.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:24:56.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:24:56.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:24:56.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:24:56.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:24:56.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:24:56.741 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:24:56.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:24:56.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:24:56.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:24:56.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:24:56.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:24:56.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:24:56.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:24:56.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:24:56.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:24:56.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:24:56.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:24:56.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:24:56.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:24:56.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:24:56.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:24:56.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:24:56.742 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:24:56.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:24:56.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:24:56.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:24:56.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/ 2024-01-13T08:24:56.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: lvm/zap.py", line 302, in zap_osd 2024-01-13T08:24:56.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:24:56.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:24:56.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:24:56.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:24:56.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:24:56.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:24:56.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: main() 2024-01-13T08:24:56.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:24:56.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: r = ctx.func(ctx) 2024-01-13T08:24:56.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:24:56.743 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:24:56.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:24:56.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:24:56.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:24:56.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:24:56.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:24:56.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:24:56.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:24:56.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:24:56.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:24:56.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:24:56.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp5oq8mucz:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbeyo9n8v:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:24:56.744 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:24:56.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T08:24:56.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: if self._apply_service(spec): 2024-01-13T08:24:56.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T08:24:56.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T08:24:56.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T08:24:56.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T08:24:56.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T08:24:56.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T08:24:56.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T08:24:56.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T08:24:56.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T08:24:56.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: raise self._value 2024-01-13T08:24:56.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T08:24:56.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: result = (True, func(*args, **kwds)) 2024-01-13T08:24:56.745 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T08:24:56.746 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: return list(map(*args)) 2024-01-13T08:24:56.746 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:24:56.746 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: return f(*arg) 2024-01-13T08:24:56.746 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:24:56.746 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:24:56.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:24:56.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: code, '\n'.join(err))) 2024-01-13T08:24:56.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:24:56.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp5oq8mucz:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbeyo9n8v:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:24:56.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:24:56.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:24:56.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:24:56.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 1b27664f-50ee-481c-845c-8da9454e9617 2024-01-13T08:24:56.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:24:56.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:24:56.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:24:56.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:24:56.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:24:56.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr self.prepare() 2024-01-13T08:24:56.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:24:56.747 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:24:56.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:24:56.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:24:56.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:24:56.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:24:56.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:24:56.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:24:56.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:24:56.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:24:56.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:24:56.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:24:56.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:24:56.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:24:56.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:24:56.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:24:56.748 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:24:56.749 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr 2024-01-13T08:24:56.749 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:24:56.749 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:24:56.749 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:24:56.749 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:24:56.749 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:24:56.749 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:24:56.749 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:24:56.749 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:24:56.749 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:24:56.749 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr instance.main() 2024-01-13T08:24:56.749 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:24:56.749 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:24:56.750 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:24:56.750 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:24:56.750 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:24:56.750 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:24:56.750 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:24:56.750 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:24:56.750 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:24:56.750 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:24:56.750 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:24:56.750 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:24:56.750 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:24:56.750 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:24:56.750 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:24:56.750 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:24:56.750 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:24:56.751 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:24:56.751 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:24:56.751 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:24:56.751 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:24:56.751 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:24:56.751 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:24:56.751 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: Traceback (most recent call last): 2024-01-13T08:24:56.751 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:24:56.751 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: main() 2024-01-13T08:24:56.751 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:24:56.751 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: r = ctx.func(ctx) 2024-01-13T08:24:56.751 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:24:56.751 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:24:56.751 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:24:56.751 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:24:56.751 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:24:56.752 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:24:56.752 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:24:56.752 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: return func(ctx) 2024-01-13T08:24:56.752 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:24:56.752 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:24:56.752 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:24:56.752 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:24:56.752 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp5oq8mucz:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbeyo9n8v:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:24:56.752 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: cluster 2024-01-13T08:24:55.608166+0000 mgr.smithi099.oodult (mgr.14180) 268 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:56.752 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: cluster 2024-01-13T08:24:55.656470+0000 mon.smithi099 (mon.0) 630 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T08:24:56.752 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:56 smithi099 conmon[33313]: audit 2024-01-13T08:24:56.272567+0000 mon.smithi099 (mon.0) 631 : audit [DBG] from='client.? 172.21.15.99:0/4017487269' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:24:56.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: cephadm 2024-01-13T08:24:55.605648+0000 mgr.smithi099.oodult (mgr.14180) 266 : cephadm [ERR] executing create_from_spec_one(([('smithi099', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[])), ('smithi131', DeviceSelection(data devices=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'], wal_devices=[], db devices=[], journal devices=[]))],)) failed. 2024-01-13T08:24:56.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:24:56.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:24:56.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: return f(*arg) 2024-01-13T08:24:56.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:24:56.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:24:56.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:24:56.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: code, '\n'.join(err))) 2024-01-13T08:24:56.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi099/config 2024-01-13T08:24:56.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpbo572j8d:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpzrvovat9:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:24:56.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:24:56.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:24:56.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:24:56.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 3d5544d4-83c1-47b6-9aa7-0f362d68e82c 2024-01-13T08:24:56.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:24:56.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:24:56.900 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:24:56.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:24:56.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:24:56.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr self.prepare() 2024-01-13T08:24:56.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:24:56.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:24:56.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:24:56.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:24:56.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:24:56.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:24:56.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:24:56.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:24:56.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:24:56.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:24:56.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:24:56.901 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:24:56.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:24:56.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:24:56.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:24:56.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:24:56.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:24:56.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:24:56.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:24:56.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:24:56.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:24:56.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:24:56.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:24:56.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:24:56.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:24:56.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:24:56.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:24:56.902 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:24:56.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:24:56.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:24:56.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:24:56.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:24:56.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:24:56.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:24:56.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:24:56.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:24:56.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:24:56.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:24:56.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:24:56.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:24:56.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:24:56.903 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:24:56.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:24:56.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:24:56.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:24:56.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:24:56.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:24:56.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:24:56.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:24:56.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:24:56.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:24:56.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:24:56.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: main() 2024-01-13T08:24:56.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:24:56.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: r = ctx.func(ctx) 2024-01-13T08:24:56.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:24:56.904 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:24:56.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:24:56.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:24:56.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:24:56.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:24:56.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-8777 2024-01-13T08:24:56.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: 4f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:24:56.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:24:56.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:24:56.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:24:56.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:24:56.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:24:56.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi099 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpbo572j8d:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpzrvovat9:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:24:56.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: cephadm 2024-01-13T08:24:55.607566+0000 mgr.smithi099.oodult (mgr.14180) 267 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd 2024-01-13T08:24:56.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: service_id: all-available-devices 2024-01-13T08:24:56.905 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: service_name: osd.all-available-devices 2024-01-13T08:24:56.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: placement: 2024-01-13T08:24:56.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: host_pattern: '*' 2024-01-13T08:24:56.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: spec: 2024-01-13T08:24:56.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: data_devices: 2024-01-13T08:24:56.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: all: true 2024-01-13T08:24:56.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: filter_logic: AND 2024-01-13T08:24:56.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: objectstore: bluestore 2024-01-13T08:24:56.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: ''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:24:56.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp5oq8mucz:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbeyo9n8v:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:24:56.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:24:56.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:24:56.906 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:24:56.907 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 1b27664f-50ee-481c-845c-8da9454e9617 2024-01-13T08:24:56.907 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:24:56.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:24:56.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:24:56.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:24:56.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:24:56.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr self.prepare() 2024-01-13T08:24:56.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:24:56.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:24:56.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:24:56.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:24:56.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:24:56.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:24:56.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:24:56.908 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:24:56.909 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:24:56.909 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:24:56.909 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:24:56.909 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:24:56.909 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:24:56.909 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:24:56.909 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:24:56.909 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:24:56.909 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:24:56.909 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:24:56.909 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:24:56.909 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:24:56.909 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:24:56.909 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:24:56.909 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:24:56.910 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:24:56.910 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:24:56.910 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:24:56.910 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:24:56.910 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:24:56.910 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:24:56.910 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:24:56.910 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:24:56.910 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:24:56.910 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:24:56.910 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:24:56.910 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:24:56.910 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:24:56.910 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:24:56.910 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:24:56.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:24:56.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:24:56.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:24:56.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:24:56.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:24:56.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:24:56.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:24:56.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/ 2024-01-13T08:24:56.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: lvm/zap.py", line 302, in zap_osd 2024-01-13T08:24:56.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:24:56.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:24:56.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:24:56.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:24:56.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:24:56.911 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:24:56.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: main() 2024-01-13T08:24:56.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:24:56.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: r = ctx.func(ctx) 2024-01-13T08:24:56.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:24:56.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:24:56.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:24:56.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:24:56.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:24:56.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:24:56.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:24:56.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:24:56.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:24:56.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:24:56.912 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:24:56.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:24:56.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp5oq8mucz:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbeyo9n8v:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:24:56.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:24:56.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 521, in _apply_all_services 2024-01-13T08:24:56.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: if self._apply_service(spec): 2024-01-13T08:24:56.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 595, in _apply_service 2024-01-13T08:24:56.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) 2024-01-13T08:24:56.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 73, in create_from_spec 2024-01-13T08:24:56.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) 2024-01-13T08:24:56.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 83, in forall_hosts_wrapper 2024-01-13T08:24:56.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) 2024-01-13T08:24:56.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map 2024-01-13T08:24:56.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: return self._map_async(func, iterable, mapstar, chunksize).get() 2024-01-13T08:24:56.913 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get 2024-01-13T08:24:56.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: raise self._value 2024-01-13T08:24:56.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker 2024-01-13T08:24:56.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: result = (True, func(*args, **kwds)) 2024-01-13T08:24:56.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar 2024-01-13T08:24:56.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: return list(map(*args)) 2024-01-13T08:24:56.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/utils.py", line 77, in do_work 2024-01-13T08:24:56.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: return f(*arg) 2024-01-13T08:24:56.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one 2024-01-13T08:24:56.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars 2024-01-13T08:24:56.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 91, in create_single_host 2024-01-13T08:24:56.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: code, '\n'.join(err))) 2024-01-13T08:24:56.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/mon.smithi131/config 2024-01-13T08:24:56.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: Non-zero exit code 1 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp5oq8mucz:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbeyo9n8v:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:24:56.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr --> passed data devices: 4 physical, 0 LVM 2024-01-13T08:24:56.914 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr --> relative data size: 1.0 2024-01-13T08:24:56.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key 2024-01-13T08:24:56.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 1b27664f-50ee-481c-845c-8da9454e9617 2024-01-13T08:24:56.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes 2024-01-13T08:24:56.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it 2024-01-13T08:24:56.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr stderr: purged osd.0 2024-01-13T08:24:56.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:24:56.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 240, in safe_prepare 2024-01-13T08:24:56.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr self.prepare() 2024-01-13T08:24:56.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:24:56.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:24:56.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 351, in prepare 2024-01-13T08:24:56.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr block_lv = self.prepare_data_device('block', osd_fsid) 2024-01-13T08:24:56.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in prepare_data_device 2024-01-13T08:24:56.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr if disk.is_partition(device) or disk.is_device(device): 2024-01-13T08:24:56.915 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/disk.py", line 373, in is_partition 2024-01-13T08:24:56.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr partitions = get_partitions() 2024-01-13T08:24:56.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr NameError: name 'get_partitions' is not defined 2024-01-13T08:24:56.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:24:56.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr During handling of the above exception, another exception occurred: 2024-01-13T08:24:56.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:24:56.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr Traceback (most recent call last): 2024-01-13T08:24:56.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in 2024-01-13T08:24:56.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() 2024-01-13T08:24:56.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ 2024-01-13T08:24:56.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr self.main(self.argv) 2024-01-13T08:24:56.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr 2024-01-13T08:24:56.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc 2024-01-13T08:24:56.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr return f(*a, **kw) 2024-01-13T08:24:56.916 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main 2024-01-13T08:24:56.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) 2024-01-13T08:24:56.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:24:56.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:24:56.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main 2024-01-13T08:24:56.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr terminal.dispatch(self.mapper, self.argv) 2024-01-13T08:24:56.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch 2024-01-13T08:24:56.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr instance.main() 2024-01-13T08:24:56.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:24:56.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:24:56.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 450, in main 2024-01-13T08:24:56.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr self._execute(plan) 2024-01-13T08:24:56.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 469, in _execute 2024-01-13T08:24:56.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr c.create(argparse.Namespace(**args)) 2024-01-13T08:24:56.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:24:56.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:24:56.917 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create 2024-01-13T08:24:56.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr prepare_step.safe_prepare(args) 2024-01-13T08:24:56.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 244, in safe_prepare 2024-01-13T08:24:56.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr rollback_osd(self.args, self.osd_id) 2024-01-13T08:24:56.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 36, in rollback_osd 2024-01-13T08:24:56.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr Zap(['--destroy', '--osd-id', osd_id]).main() 2024-01-13T08:24:56.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main 2024-01-13T08:24:56.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr self.zap_osd() 2024-01-13T08:24:56.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root 2024-01-13T08:24:56.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr return func(*a, **kw) 2024-01-13T08:24:56.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 302, in zap_osd 2024-01-13T08:24:56.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) 2024-01-13T08:24:56.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices 2024-01-13T08:24:56.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr '%s' % osd_id or osd_fsid) 2024-01-13T08:24:56.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: /bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 0 2024-01-13T08:24:56.918 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: Traceback (most recent call last): 2024-01-13T08:24:56.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9509, in 2024-01-13T08:24:56.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: main() 2024-01-13T08:24:56.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 9497, in main 2024-01-13T08:24:56.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: r = ctx.func(ctx) 2024-01-13T08:24:56.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2009, in _infer_config 2024-01-13T08:24:56.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:24:56.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1940, in _infer_fsid 2024-01-13T08:24:56.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:24:56.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 2037, in _infer_image 2024-01-13T08:24:56.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:24:56.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1927, in _validate_fsid 2024-01-13T08:24:56.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: return func(ctx) 2024-01-13T08:24:56.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 5554, in command_ceph_volume 2024-01-13T08:24:56.919 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: out, err, code = call_throws(ctx, c.run_cmd()) 2024-01-13T08:24:56.920 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: File "/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/cephadm.cf1c9b864d29dd2645bb517833464f26027d435bb39febbf565308fda83f90d2", line 1720, in call_throws 2024-01-13T08:24:56.920 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-13T08:24:56.920 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: RuntimeError: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 -e NODE_NAME=smithi131 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/run/ceph:z -v /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp5oq8mucz:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpbeyo9n8v:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b926bb9e029d1391c1bd2df0fbc0e8b591effa2f2f7ee12facc20da759af9755 lvm batch --no-auto /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 --yes --no-systemd 2024-01-13T08:24:56.920 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: cluster 2024-01-13T08:24:55.608166+0000 mgr.smithi099.oodult (mgr.14180) 268 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:56.920 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: cluster 2024-01-13T08:24:55.656470+0000 mon.smithi099 (mon.0) 630 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.all-available-devices (CEPHADM_APPLY_SPEC_FAIL) 2024-01-13T08:24:56.920 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:56 smithi131 conmon[38024]: audit 2024-01-13T08:24:56.272567+0000 mon.smithi099 (mon.0) 631 : audit [DBG] from='client.? 172.21.15.99:0/4017487269' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:24:56.975 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:24:57.976 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:24:58.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:24:58 smithi099 conmon[33313]: cluster 2024-01-13T08:24:57.608724+0000 mgr.smithi099.oodult (mgr.14180) 269 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:59.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:58 smithi131 conmon[38024]: cluster 2024-01-13T08:24:57.608724+0000 mgr.smithi099.oodult (mgr.14180) 269 : cluster [DBG] 2024-01-13T08:24:59.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:24:58 smithi131 conmon[38024]: pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:24:59.700 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:25:00.256 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:25:00.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:00 smithi099 conmon[33313]: cluster 2024-01-13T08:24:59.609134+0000 mgr.smithi099.oodult (mgr.14180) 270 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:00.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:00 smithi099 conmon[33313]: audit 2024-01-13T08:24:59.703745+0000 mon.smithi099 (mon.0) 632 : audit [DBG] from='client.? 172.21.15.99:0/2049891248' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:25:01.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:00 smithi131 conmon[38024]: cluster 2024-01-13T08:24:59.609134+0000 mgr.smithi099.oodult (mgr.14180) 270 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:01.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:00 smithi131 conmon[38024]: audit 2024-01-13T08:25:01.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:00 smithi131 conmon[38024]: 2024-01-13T08:24:59.703745+0000 mon.smithi099 (mon.0) 632 : audit [DBG] from='client.? 172.21.15.99:0/2049891248' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:25:01.257 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:25:02.354 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:02 smithi099 conmon[33313]: cluster 2024-01-13T08:25:01.609543+0000 mgr.smithi099.oodult (mgr.14180) 271 2024-01-13T08:25:02.355 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:02 smithi099 conmon[33313]: : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:02.398 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:02 smithi131 conmon[38024]: cluster 2024-01-13T08:25:01.609543+0000 mgr.smithi099.oodult (mgr.14180) 271 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:02.398 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:02 smithi131 conmon[38024]: 2024-01-13T08:25:02.934 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:25:03.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:03 smithi099 conmon[33313]: audit 2024-01-13T08:25:02.937295+0000 mon.smithi099 (mon.0) 633 : audit [DBG] from='client.? 172.21.15.99:0/3280339483' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:25:03.398 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:03 smithi131 conmon[38024]: audit 2024-01-13T08:25:02.937295+0000 mon.smithi099 (mon.0) 633 : audit 2024-01-13T08:25:03.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:03 smithi131 conmon[38024]: [DBG] from='client.? 172.21.15.99:0/3280339483' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:25:03.480 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:25:04.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:04 smithi131 conmon[38024]: cluster 2024-01-13T08:25:03.610062+0000 mgr.smithi099.oodult (mgr.14180) 272 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:04.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:04 smithi099 conmon[33313]: cluster 2024-01-13T08:25:03.610062+0000 mgr.smithi099.oodult (mgr.14180) 272 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:04.481 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:25:06.244 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:25:06.723 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:25:06.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:06 smithi099 conmon[33313]: cluster 2024-01-13T08:25:05.610416+0000 mgr.smithi099.oodult (mgr.14180) 273 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:06.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:06 smithi099 conmon[33313]: audit 2024-01-13T08:25:06.248520+0000 mon.smithi099 (mon.0) 634 : audit [DBG] from='client.? 172.21.15.99:0/4237034801' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:25:07.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:06 smithi131 conmon[38024]: cluster 2024-01-13T08:25:05.610416+0000 mgr.smithi099.oodult (mgr.14180) 273 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:07.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:06 smithi131 conmon[38024]: audit 2024-01-13T08:25:06.248520+0000 mon.smithi099 (mon.0) 634 : audit [DBG] from='client.? 172.21.15.99:0/4237034801' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:25:07.723 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:25:08.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:08 smithi099 conmon[33313]: audit 2024-01-13T08:25:07.251864+0000 mon.smithi099 (mon.0) 635 : 2024-01-13T08:25:08.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:08 smithi099 conmon[33313]: audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:25:08.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:08 smithi099 conmon[33313]: cluster 2024-01-13T08:25:07.610863+0000 mgr.smithi099.oodult (mgr.14180) 274 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:08.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:08 smithi131 conmon[38024]: audit 2024-01-13T08:25:07.251864+0000 mon.smithi099 ( 2024-01-13T08:25:08.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:08 smithi131 conmon[38024]: mon.0) 635 : audit [INF] from='mgr.14180 172.21.15.99:0/4092507245' entity='mgr.smithi099.oodult' 2024-01-13T08:25:08.649 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:08 smithi131 conmon[38024]: cluster 2024-01-13T08:25:07.610863+0000 mgr.smithi099.oodult (mgr.14180) 274 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:09.485 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:25:09.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:09 smithi131 conmon[38024]: audit 2024-01-13T08:25:09.489205+0000 mon.smithi099 (mon.0) 636 : audit [DBG] from='client.? 172.21.15.99:0/670353055' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:25:09.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:09 smithi099 conmon[33313]: audit 2024-01-13T08:25:09.489205+0000 mon.smithi099 (mon.0) 636 : audit [DBG] from='client.? 172.21.15.99:0/670353055' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:25:10.029 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:25:10.899 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:10 smithi131 conmon[38024]: cluster 2024-01-13T08:25:09.611317+0000 mgr.smithi099.oodult (mgr.14180) 275 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:10.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:10 smithi099 conmon[33313]: cluster 2024-01-13T08:25:09.611317+0000 mgr.smithi099.oodult (mgr.14180) 275 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:11.029 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:25:12.321 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:12 smithi099 conmon[33313]: cluster 2024-01-13T08:25:11.611668+0000 mgr.smithi099.oodult (mgr.14180) 276 : cluster 2024-01-13T08:25:12.321 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:12 smithi099 conmon[33313]: [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:12.398 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:12 smithi131 conmon[38024]: cluster 2024-01-13T08:25:11.611668+0000 mgr.smithi099.oodult (mgr.14180) 276 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:12.673 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:25:13.195 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:25:13.398 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:13 smithi131 conmon[38024]: audit 2024-01-13T08:25:12.677066+0000 mon.smithi099 (mon.0) 637 : audit [DBG] from='client.? 172.21.15.99:0/3097930783' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:25:13.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:13 smithi099 conmon[33313]: audit 2024-01-13T08:25:12.677066+0000 mon.smithi099 (mon.0) 637 : audit [DBG] from='client.? 172.21.15.99:0/3097930783' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:25:14.196 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:25:14.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:14 smithi131 conmon[38024]: cluster 2024-01-13T08:25:13.612119+0000 mgr.smithi099.oodult (mgr.14180) 277 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:14.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:14 smithi099 conmon[33313]: cluster 2024-01-13T08:25:13.612119+0000 mgr.smithi099.oodult (mgr.14180) 277 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:15.905 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:25:16.443 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:25:16.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:16 smithi099 conmon[33313]: cluster 2024-01-13T08:25:15.612539+0000 mgr.smithi099.oodult (mgr.14180) 278 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:16.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:16 smithi099 conmon[33313]: audit 2024-01-13T08:25:15.909288+0000 mon.smithi099 (mon.0) 638 : audit [DBG] from='client.? 172.21.15.99:0/379472477' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:25:17.148 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:16 smithi131 conmon[38024]: cluster 2024-01-13T08:25:15.612539+0000 mgr.smithi099.oodult (mgr.14180) 278 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:17.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:16 smithi131 conmon[38024]: audit 2024-01-13T08:25:17.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:16 smithi131 conmon[38024]: 2024-01-13T08:25:15.909288+0000 mon.smithi099 (mon.0) 638 : audit [DBG] from='client.? 172.21.15.99:0/379472477' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:25:17.444 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:25:18.828 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:18 smithi099 conmon[33313]: cluster 2024-01-13T08:25:17.613006+0000 mgr.smithi099.oodult (mgr.14180) 279 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:19.141 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:25:19.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:18 smithi131 conmon[38024]: cluster 2024-01-13T08:25:17.613006+0000 mgr.smithi099.oodult (mgr.14180) 279 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:19.678 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:25:19.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:19 smithi099 conmon[33313]: audit 2024-01-13T08:25:19.145502+0000 mon.smithi099 (mon.0) 639 : audit [DBG] from='client.? 172.21.15.99:0/1744170796' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:25:20.148 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:19 smithi131 conmon[38024]: audit 2024-01-13T08:25:19.145502+0000 mon.smithi099 (mon.0) 639 : audit [DBG] from='client.? 172.21.15.99:0/1744170796' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:25:20.679 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:25:20.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:20 smithi099 conmon[33313]: cluster 2024-01-13T08:25:19.613428+0000 mgr.smithi099.oodult (mgr.14180) 280 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:21.148 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:20 smithi131 conmon[38024]: cluster 2024-01-13T08:25:19.613428+0000 mgr.smithi099.oodult (mgr.14180) 280 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:22.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:22 smithi131 conmon[38024]: cluster 2024-01-13T08:25:21.613790+0000 mgr.smithi099.oodult (mgr.14180) 281 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:22.421 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:25:22.441 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:22 smithi099 conmon[33313]: cluster 2024-01-13T08:25:21.613790+0000 mgr.smithi099.oodult (mgr.14180) 281 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:22.982 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:25:23.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:23 smithi099 conmon[33313]: audit 2024-01-13T08:25:22.424441+0000 mon.smithi099 (mon.0) 640 : audit [DBG] from='client.? 172.21.15.99:0/2647012797' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:25:23.398 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:23 smithi131 conmon[38024]: audit 2024-01-13T08:25:22.424441+0000 mon.smithi099 (mon.0) 640 : audit [DBG] from='client.? 172.21.15.99:0/2647012797' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:25:23.983 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:25:24.398 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:24 smithi131 conmon[38024]: cluster 2024-01-13T08:25:23.614287+0000 mgr.smithi099.oodult (mgr.14180) 282 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:24.459 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:24 smithi099 conmon[33313]: cluster 2024-01-13T08:25:23.614287+0000 mgr.smithi099.oodult (mgr.14180) 282 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:25.755 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:25:26.326 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:25:26.982 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:26 smithi099 conmon[33313]: cluster 2024-01-13T08:25:25.614642+0000 mgr.smithi099.oodult (mgr.14180) 283 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:26.983 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:26 smithi099 conmon[33313]: audit 2024-01-13T08:25:25.759760+0000 mon.smithi099 (mon.0) 641 : audit [DBG] from='client.? 172.21.15.99:0/417912959' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:25:27.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:26 smithi131 conmon[38024]: cluster 2024-01-13T08:25:25.614642+0000 mgr.smithi099.oodult (mgr.14180) 283 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:27.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:26 smithi131 conmon[38024]: audit 2024-01-13T08:25:27.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:26 smithi131 conmon[38024]: 2024-01-13T08:25:25.759760+0000 mon.smithi099 (mon.0) 641 : audit [DBG] from='client.? 172.21.15.99:0/417912959' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:25:27.327 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:25:28.728 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:28 smithi099 conmon[33313]: cluster 2024-01-13T08:25:27.615018+0000 mgr.smithi099.oodult (mgr.14180) 284 : cluster 2024-01-13T08:25:28.728 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:28 smithi099 conmon[33313]: [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:29.035 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:25:29.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:28 smithi131 conmon[38024]: cluster 2024-01-13T08:25:27.615018+0000 mgr.smithi099.oodult (mgr.14180) 284 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:29.608 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:25:29.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:29 smithi099 conmon[33313]: audit 2024-01-13T08:25:29.039636+0000 mon.smithi099 (mon.0) 642 : audit [DBG] from='client.? 172.21.15.99:0/2606014894' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:25:30.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:29 smithi131 conmon[38024]: audit 2024-01-13T08:25:29.039636+0000 mon.smithi099 (mon.0) 642 : audit [DBG] from='client.? 172.21.15.99:0/2606014894' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:25:30.610 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:25:30.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:30 smithi099 conmon[33313]: cluster 2024-01-13T08:25:29.615662+0000 mgr.smithi099.oodult (mgr.14180) 285 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:31.148 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:30 smithi131 conmon[38024]: cluster 2024-01-13T08:25:29.615662+0000 mgr.smithi099.oodult (mgr.14180) 285 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:32.345 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:25:32.370 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:32 smithi099 conmon[33313]: cluster 2024-01-13T08:25:31.616306+0000 mgr.smithi099.oodult (mgr.14180) 286 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:32.398 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:32 smithi131 conmon[38024]: cluster 2024-01-13T08:25:31.616306+0000 mgr.smithi099.oodult (mgr.14180) 286 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:32.866 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:25:33.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:33 smithi099 conmon[33313]: audit 2024-01-13T08:25:32.349021+0000 mon.smithi099 (mon.0) 643 : audit [DBG] from='client.? 172.21.15.99:0/7914411' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:25:33.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:33 smithi131 conmon[38024]: audit 2024-01-13T08:25:32.349021+0000 mon.smithi099 (mon.0) 643 : audit [DBG] from='client.? 172.21.15.99:0/7914411' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:25:33.399 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:33 smithi131 conmon[38024]: 2024-01-13T08:25:33.867 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:25:34.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:34 smithi099 conmon[33313]: cluster 2024-01-13T08:25:33.616706+0000 mgr.smithi099.oodult (mgr. 2024-01-13T08:25:34.231 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:34 smithi099 conmon[33313]: 14180) 287 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:34.398 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:34 smithi131 conmon[38024]: cluster 2024-01-13T08:25:33.616706+0000 mgr.smithi099.oodult (mgr.14180) 287 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:35.611 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:25:35.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:35 smithi099 conmon[33313]: audit 2024-01-13T08:25:35.615933+0000 mon.smithi099 (mon. 2024-01-13T08:25:35.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:35 smithi099 conmon[33313]: 0) 644 : audit [DBG] from='client.? 172.21.15.99:0/897062761' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:25:36.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:35 smithi131 conmon[38024]: audit 2024-01-13T08:25:35.615933+0000 mon.smithi099 (mon.0) 644 : audit [DBG] from='client.? 172.21.15.99:0/897062761' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:25:36.164 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:25:36.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:36 smithi099 conmon[33313]: cluster 2024-01-13T08:25:35.617129+0000 mgr.smithi099.oodult (mgr.14180 2024-01-13T08:25:36.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:36 smithi099 conmon[33313]: ) 288 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:37.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:36 smithi131 conmon[38024]: cluster 2024-01-13T08:25:35.617129+0000 mgr.smithi099.oodult (mgr.14180) 288 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:37.165 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:25:38.926 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:25:38.947 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:38 smithi099 conmon[33313]: cluster 2024-01-13T08:25:37.617596+0000 mgr.smithi099.oodult (mgr.14180) 289 : cluster 2024-01-13T08:25:38.947 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:38 smithi099 conmon[33313]: [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:39.148 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:38 smithi131 conmon[38024]: cluster 2024-01-13T08:25:37.617596+0000 mgr.smithi099.oodult (mgr.14180) 289 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:39.441 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:25:39.981 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:39 smithi099 conmon[33313]: audit 2024-01-13T08:25:38.930450+0000 mon.smithi099 (mon.0) 645 : audit [DBG] from='client.? 172.21.15.99:0/161117958' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:25:40.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:39 smithi131 conmon[38024]: audit 2024-01-13T08:25:38.930450+0000 mon.smithi099 (mon.0) 645 : audit [DBG] from='client.? 172.21.15.99:0/161117958' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:25:40.441 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:445562ab4bc3ddfb386936119050695810860bcb shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 -- ceph osd stat -f json 2024-01-13T08:25:40.731 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:40 smithi099 conmon[33313]: cluster 2024-01-13T08:25:39.618058+0000 mgr.smithi099.oodult (mgr.14180) 290 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:41.148 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:40 smithi131 conmon[38024]: cluster 2024-01-13T08:25:39.618058+0000 mgr.smithi099.oodult (mgr.14180) 290 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:42.188 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-01-13T08:25:42.355 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:42 smithi099 conmon[33313]: cluster 2024-01-13T08:25:41.618551+0000 mgr.smithi099.oodult (mgr.14180) 291 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:42.398 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:42 smithi131 conmon[38024]: cluster 2024-01-13T08:25:41.618551+0000 mgr.smithi099.oodult (mgr.14180) 291 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-13T08:25:42.756 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":33,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":1705134035,"num_remapped_pgs":0} 2024-01-13T08:25:42.757 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 779, in ceph_osds while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (121) after waiting for 120 seconds 2024-01-13T08:25:42.758 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-13T08:25:42.789 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-13T08:25:42.843 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-01-13T08:25:42.843 DEBUG:teuthology.orchestra.run.smithi099:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-13T08:25:42.861 DEBUG:teuthology.orchestra.run.smithi131:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-13T08:25:42.879 INFO:tasks.cephadm:Stopping all daemons... 2024-01-13T08:25:42.879 INFO:tasks.cephadm.mon.smithi099:Stopping mon.smithi099... 2024-01-13T08:25:42.879 DEBUG:teuthology.orchestra.run.smithi099:> sudo systemctl stop ceph-f1ba87a2-b1eb-11ee-95ac-87774f69a715@mon.smithi099 2024-01-13T08:25:43.078 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:42 smithi099 systemd[1]: Stopping Ceph mon.smithi099 for f1ba87a2-b1eb-11ee-95ac-87774f69a715... 2024-01-13T08:25:43.078 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:42 smithi099 bash[100755]: Error: no container with name or ID ceph-f1ba87a2-b1eb-11ee-95ac-87774f69a715-mon.smithi099 found: no such container 2024-01-13T08:25:43.149 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:43 smithi131 conmon[38024]: audit 2024-01-13T08:25:42.192065+0000 mon.smithi099 (mon.0) 646 : audit [DBG] from='client.? 172.21.15.99:0/980622598' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:25:43.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:43 smithi099 conmon[33313]: audit 2024-01-13T08:25:42.192065+0000 mon.smithi099 2024-01-13T08:25:43.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:43 smithi099 conmon[33313]: (mon.0) 646 : audit [DBG] from='client.? 172.21.15.99:0/980622598' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-13T08:25:43.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:43 smithi099 conmon[33313]: debug 2024-01-13T08:25:43.200+0000 7f68f5d4e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.smithi099 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-13T08:25:43.481 INFO:journalctl@ceph.mon.smithi099.smithi099.stdout:Jan 13 08:25:43 smithi099 conmon[33313]: debug 2024-01-13T08:25:43.200+0000 7f68f5d4e700 -1 mon.smithi099@0(leader) e2 *** Got Signal Terminated *** 2024-01-13T08:25:43.819 DEBUG:teuthology.orchestra.run.smithi099:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f1ba87a2-b1eb-11ee-95ac-87774f69a715@mon.smithi099.service' 2024-01-13T08:25:43.870 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-13T08:25:43.870 INFO:tasks.cephadm.mon.smithi099:Stopped mon.smithi099 2024-01-13T08:25:43.871 INFO:tasks.cephadm.mon.smithi131:Stopping mon.smithi131... 2024-01-13T08:25:43.871 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-f1ba87a2-b1eb-11ee-95ac-87774f69a715@mon.smithi131 2024-01-13T08:25:44.173 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:43 smithi131 systemd[1]: Stopping Ceph mon.smithi131 for f1ba87a2-b1eb-11ee-95ac-87774f69a715... 2024-01-13T08:25:44.174 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:43 smithi131 bash[45534]: Error: no container with name or ID ceph-f1ba87a2-b1eb-11ee-95ac-87774f69a715-mon.smithi131 found: no such container 2024-01-13T08:25:44.478 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:44 smithi131 conmon[38024]: debug 2024-01-13T08:25:44.176+0000 7f4a1d31b700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.smithi131 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-13T08:25:44.478 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:44 smithi131 conmon[38024]: debug 2024-01-13T08:25:44.176+0000 7f4a1d31b700 -1 mon.smithi131@1(peon) e2 *** Got Signal Terminated *** 2024-01-13T08:25:44.898 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Jan 13 08:25:44 smithi131 bash[45649]: b7c0aa4e00c3126475b93d64d82c907685af629c4fccd90907a64c12edfe60e5 2024-01-13T08:25:45.031 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f1ba87a2-b1eb-11ee-95ac-87774f69a715@mon.smithi131.service' 2024-01-13T08:25:45.075 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-13T08:25:45.075 INFO:tasks.cephadm.mon.smithi131:Stopped mon.smithi131 2024-01-13T08:25:45.076 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 --force --keep-logs 2024-01-13T08:25:51.572 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 --force --keep-logs 2024-01-13T08:25:55.279 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-13T08:25:55.317 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-13T08:25:55.347 INFO:tasks.cephadm:Archiving crash dumps... 2024-01-13T08:25:55.349 DEBUG:teuthology.misc:Transferring archived files from smithi099:/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515725/remote/ubuntu@smithi099.front.sepia.ceph.com/crash 2024-01-13T08:25:55.350 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar c -f - -C /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash -- . 2024-01-13T08:25:55.408 INFO:teuthology.orchestra.run.smithi099.stderr:tar: /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash: Cannot open: No such file or directory 2024-01-13T08:25:55.409 INFO:teuthology.orchestra.run.smithi099.stderr:tar: Error is not recoverable: exiting now 2024-01-13T08:25:55.410 DEBUG:teuthology.misc:Transferring archived files from smithi131:/var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515725/remote/ubuntu@smithi131.front.sepia.ceph.com/crash 2024-01-13T08:25:55.411 DEBUG:teuthology.orchestra.run.smithi131:> sudo tar c -f - -C /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash -- . 2024-01-13T08:25:55.458 INFO:teuthology.orchestra.run.smithi131.stderr:tar: /var/lib/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/crash: Cannot open: No such file or directory 2024-01-13T08:25:55.458 INFO:teuthology.orchestra.run.smithi131.stderr:tar: Error is not recoverable: exiting now 2024-01-13T08:25:55.459 INFO:tasks.cephadm:Checking cluster log for badness... 2024-01-13T08:25:55.459 DEBUG:teuthology.orchestra.run.smithi099:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-01-13T08:25:55.489 INFO:teuthology.orchestra.run.smithi099.stderr:grep: /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/ceph.log: No such file or directory 2024-01-13T08:25:55.491 INFO:tasks.cephadm:Compressing logs... 2024-01-13T08:25:55.491 DEBUG:teuthology.orchestra.run.smithi099:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-13T08:25:55.534 DEBUG:teuthology.orchestra.run.smithi131:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-13T08:25:55.560 INFO:teuthology.orchestra.run.smithi099.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-01-13T08:25:55.561 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-01-13T08:25:55.562 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/ceph-volume.log 2024-01-13T08:25:55.563 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/ceph-mgr.smithi099.oodult.log 2024-01-13T08:25:55.563 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/ceph-mon.smithi099.log 2024-01-13T08:25:55.563 INFO:teuthology.orchestra.run.smithi131.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-01-13T08:25:55.563 INFO:teuthology.orchestra.run.smithi131.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-01-13T08:25:55.563 INFO:teuthology.orchestra.run.smithi131.stderr:gzip -5 --verbose -- /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/ceph-mgr.smithi131.qnazik.log 2024-01-13T08:25:55.563 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/ceph-mon.smithi131.log 2024-01-13T08:25:55.563 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/ceph-mgr.smithi131.qnazik.log: gzip -5 --verbose -- /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/ceph-volume.log 2024-01-13T08:25:55.565 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/ceph-mon.smithi131.log: /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/ceph-volume.log: 91.7% -- replaced with /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/ceph-mgr.smithi131.qnazik.log.gz 2024-01-13T08:25:55.565 INFO:teuthology.orchestra.run.smithi131.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-01-13T08:25:55.566 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/ceph-mgr.smithi099.oodult.log: /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/ceph-mon.smithi099.log: 93.9% -- replaced with /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/ceph-volume.log.gz 2024-01-13T08:25:55.568 INFO:teuthology.orchestra.run.smithi131.stderr: 94.0% -- replaced with /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/ceph-volume.log.gz 2024-01-13T08:25:55.570 INFO:teuthology.orchestra.run.smithi099.stderr: 92.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-01-13T08:25:55.632 INFO:teuthology.orchestra.run.smithi131.stderr: 92.7% -- replaced with /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/ceph-mon.smithi131.log.gz 2024-01-13T08:25:55.633 INFO:teuthology.orchestra.run.smithi131.stderr: 2024-01-13T08:25:55.633 INFO:teuthology.orchestra.run.smithi131.stderr:real 0m0.083s 2024-01-13T08:25:55.634 INFO:teuthology.orchestra.run.smithi131.stderr:user 0m0.091s 2024-01-13T08:25:55.634 INFO:teuthology.orchestra.run.smithi131.stderr:sys 0m0.014s 2024-01-13T08:25:55.642 INFO:teuthology.orchestra.run.smithi099.stderr: 91.2% -- replaced with /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/ceph-mgr.smithi099.oodult.log.gz 2024-01-13T08:25:55.793 INFO:teuthology.orchestra.run.smithi099.stderr: 90.0% -- replaced with /var/log/ceph/f1ba87a2-b1eb-11ee-95ac-87774f69a715/ceph-mon.smithi099.log.gz 2024-01-13T08:25:55.795 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-01-13T08:25:55.795 INFO:teuthology.orchestra.run.smithi099.stderr:real 0m0.246s 2024-01-13T08:25:55.795 INFO:teuthology.orchestra.run.smithi099.stderr:user 0m0.324s 2024-01-13T08:25:55.795 INFO:teuthology.orchestra.run.smithi099.stderr:sys 0m0.026s 2024-01-13T08:25:55.795 INFO:tasks.cephadm:Archiving logs... 2024-01-13T08:25:55.796 DEBUG:teuthology.misc:Transferring archived files from smithi099:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515725/remote/ubuntu@smithi099.front.sepia.ceph.com/log 2024-01-13T08:25:55.796 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-13T08:25:55.925 DEBUG:teuthology.misc:Transferring archived files from smithi131:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515725/remote/ubuntu@smithi131.front.sepia.ceph.com/log 2024-01-13T08:25:55.926 DEBUG:teuthology.orchestra.run.smithi131:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-13T08:25:55.969 INFO:tasks.cephadm:Removing cluster... 2024-01-13T08:25:55.970 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 --force 2024-01-13T08:25:56.412 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f1ba87a2-b1eb-11ee-95ac-87774f69a715 --force 2024-01-13T08:25:56.850 INFO:tasks.cephadm:Removing cephadm ... 2024-01-13T08:25:56.851 DEBUG:teuthology.orchestra.run.smithi099:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-13T08:25:56.869 DEBUG:teuthology.orchestra.run.smithi131:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-13T08:25:56.886 INFO:tasks.cephadm:Teardown complete 2024-01-13T08:25:56.886 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 779, in ceph_osds while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (121) after waiting for 120 seconds 2024-01-13T08:25:57.097 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=8b07fd114bf845febed910225c1b776c Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_445562ab4bc3ddfb386936119050695810860bcb/qa/tasks/cephadm.py", line 779, in ceph_osds while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (121) after waiting for 120 seconds 2024-01-13T08:25:57.099 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-01-13T08:25:57.109 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop 2024-01-13T08:25:57.148 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi099:/dev/vg_nvme/lv_1... 2024-01-13T08:25:57.149 DEBUG:teuthology.orchestra.run.smithi099:> sudo nvme disconnect -n lv_1 2024-01-13T08:25:57.299 INFO:teuthology.orchestra.run.smithi099.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-01-13T08:25:57.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T08:25:57.301 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi099:/dev/vg_nvme/lv_2... 2024-01-13T08:25:57.302 DEBUG:teuthology.orchestra.run.smithi099:> sudo nvme disconnect -n lv_2 2024-01-13T08:25:57.432 INFO:teuthology.orchestra.run.smithi099.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-01-13T08:25:57.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T08:25:57.434 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi099:/dev/vg_nvme/lv_3... 2024-01-13T08:25:57.434 DEBUG:teuthology.orchestra.run.smithi099:> sudo nvme disconnect -n lv_3 2024-01-13T08:25:57.559 INFO:teuthology.orchestra.run.smithi099.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-01-13T08:25:57.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T08:25:57.560 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi099:/dev/vg_nvme/lv_4... 2024-01-13T08:25:57.561 DEBUG:teuthology.orchestra.run.smithi099:> sudo nvme disconnect -n lv_4 2024-01-13T08:25:57.682 INFO:teuthology.orchestra.run.smithi099.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-01-13T08:25:57.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T08:25:57.684 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-01-13T08:25:57.684 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/scratch_devs 2024-01-13T08:25:57.714 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi131:/dev/vg_nvme/lv_1... 2024-01-13T08:25:57.715 DEBUG:teuthology.orchestra.run.smithi131:> sudo nvme disconnect -n lv_1 2024-01-13T08:25:57.859 INFO:teuthology.orchestra.run.smithi131.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-01-13T08:25:57.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T08:25:57.861 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi131:/dev/vg_nvme/lv_2... 2024-01-13T08:25:57.861 DEBUG:teuthology.orchestra.run.smithi131:> sudo nvme disconnect -n lv_2 2024-01-13T08:25:57.981 INFO:teuthology.orchestra.run.smithi131.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-01-13T08:25:57.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T08:25:57.983 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi131:/dev/vg_nvme/lv_3... 2024-01-13T08:25:57.983 DEBUG:teuthology.orchestra.run.smithi131:> sudo nvme disconnect -n lv_3 2024-01-13T08:25:58.115 INFO:teuthology.orchestra.run.smithi131.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-01-13T08:25:58.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T08:25:58.117 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi131:/dev/vg_nvme/lv_4... 2024-01-13T08:25:58.117 DEBUG:teuthology.orchestra.run.smithi131:> sudo nvme disconnect -n lv_4 2024-01-13T08:25:58.237 INFO:teuthology.orchestra.run.smithi131.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-01-13T08:25:58.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T08:25:58.239 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-01-13T08:25:58.239 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/scratch_devs 2024-01-13T08:25:58.269 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-13T08:25:58.279 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-13T08:25:58.279 DEBUG:teuthology.orchestra.run.smithi099:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-13T08:25:58.282 DEBUG:teuthology.orchestra.run.smithi131:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-13T08:25:58.299 INFO:teuthology.orchestra.run.smithi099.stderr:bash: ntpq: command not found 2024-01-13T08:25:58.306 INFO:teuthology.orchestra.run.smithi099.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-13T08:25:58.306 INFO:teuthology.orchestra.run.smithi099.stdout:=============================================================================== 2024-01-13T08:25:58.306 INFO:teuthology.orchestra.run.smithi099.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 53 -793us[ -793us] +/- 84ms 2024-01-13T08:25:58.306 INFO:teuthology.orchestra.run.smithi099.stdout:^* hv02.front.sepia.ceph.com 3 6 377 54 +413us[ +417us] +/- 61ms 2024-01-13T08:25:58.306 INFO:teuthology.orchestra.run.smithi099.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 54 +163us[ +167us] +/- 70ms 2024-01-13T08:25:58.307 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T08:25:58.326 INFO:teuthology.orchestra.run.smithi131.stderr:bash: ntpq: command not found 2024-01-13T08:25:58.331 INFO:teuthology.orchestra.run.smithi131.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-13T08:25:58.331 INFO:teuthology.orchestra.run.smithi131.stdout:=============================================================================== 2024-01-13T08:25:58.332 INFO:teuthology.orchestra.run.smithi131.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 53 -791us[ -791us] +/- 84ms 2024-01-13T08:25:58.332 INFO:teuthology.orchestra.run.smithi131.stdout:^* hv02.front.sepia.ceph.com 3 6 377 54 +448us[ +458us] +/- 61ms 2024-01-13T08:25:58.332 INFO:teuthology.orchestra.run.smithi131.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 52 +203us[ +203us] +/- 70ms 2024-01-13T08:25:58.332 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-13T08:25:58.333 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-13T08:25:58.342 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-13T08:25:58.343 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-13T08:25:58.376 DEBUG:teuthology.orchestra.run.smithi099:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-13T08:25:58.447 DEBUG:teuthology.orchestra.run.smithi131:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-13T08:25:58.513 DEBUG:teuthology.orchestra.run.smithi099:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-13T08:25:58.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T08:25:58.546 DEBUG:teuthology.orchestra.run.smithi131:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-13T08:25:58.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T08:25:58.580 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-13T08:25:58.589 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-13T08:25:58.614 INFO:teuthology.task.internal:Duration was 1184.360494 seconds 2024-01-13T08:25:58.615 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-13T08:25:58.649 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-13T08:25:58.649 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-13T08:25:58.652 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-13T08:25:58.700 INFO:teuthology.orchestra.run.smithi131.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-13T08:25:58.700 INFO:teuthology.orchestra.run.smithi099.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-13T08:25:58.992 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-13T08:25:58.992 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi099.front.sepia.ceph.com 2024-01-13T08:25:58.992 DEBUG:teuthology.orchestra.run.smithi099:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-13T08:25:59.016 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi131.front.sepia.ceph.com 2024-01-13T08:25:59.016 DEBUG:teuthology.orchestra.run.smithi131:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-13T08:25:59.039 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-13T08:25:59.039 DEBUG:teuthology.orchestra.run.smithi099:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-13T08:25:59.059 DEBUG:teuthology.orchestra.run.smithi131:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-13T08:25:59.131 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-13T08:25:59.131 DEBUG:teuthology.orchestra.run.smithi099:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-13T08:25:59.133 DEBUG:teuthology.orchestra.run.smithi131:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-13T08:25:59.276 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-13T08:25:59.286 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-13T08:25:59.286 DEBUG:teuthology.orchestra.run.smithi099:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-13T08:25:59.313 DEBUG:teuthology.orchestra.run.smithi131:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-13T08:25:59.340 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-13T08:25:59.350 DEBUG:teuthology.orchestra.run.smithi099:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-13T08:25:59.356 DEBUG:teuthology.orchestra.run.smithi131:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-13T08:25:59.380 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern = core 2024-01-13T08:25:59.406 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern = core 2024-01-13T08:25:59.439 DEBUG:teuthology.orchestra.run.smithi099:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-13T08:25:59.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T08:25:59.458 DEBUG:teuthology.orchestra.run.smithi131:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-13T08:25:59.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-13T08:25:59.495 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-13T08:25:59.505 INFO:teuthology.task.internal:Transferring archived files... 2024-01-13T08:25:59.506 DEBUG:teuthology.misc:Transferring archived files from smithi099:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515725/remote/smithi099 2024-01-13T08:25:59.507 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-13T08:25:59.551 DEBUG:teuthology.misc:Transferring archived files from smithi131:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515725/remote/smithi131 2024-01-13T08:25:59.552 DEBUG:teuthology.orchestra.run.smithi131:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-13T08:25:59.591 INFO:teuthology.task.internal:Removing archive directory... 2024-01-13T08:25:59.591 DEBUG:teuthology.orchestra.run.smithi099:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-13T08:25:59.593 DEBUG:teuthology.orchestra.run.smithi131:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-13T08:25:59.641 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-13T08:25:59.653 INFO:teuthology.task.internal:Not uploading archives. 2024-01-13T08:25:59.653 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-13T08:25:59.682 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-13T08:25:59.682 DEBUG:teuthology.orchestra.run.smithi099:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-13T08:25:59.684 DEBUG:teuthology.orchestra.run.smithi131:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-13T08:25:59.698 INFO:teuthology.orchestra.run.smithi099.stdout: 262175 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 13 08:25 /home/ubuntu/cephtest 2024-01-13T08:25:59.700 INFO:teuthology.orchestra.run.smithi131.stdout: 262157 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 13 08:25 /home/ubuntu/cephtest 2024-01-13T08:25:59.702 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-13T08:25:59.715 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-13T08:25:59.782 INFO:teuthology.nuke:Checking targets against current locks 2024-01-13T08:25:59.808 DEBUG:teuthology.nuke:shortname: smithi099 2024-01-13T08:25:59.811 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-13T08:25:59.834 DEBUG:teuthology.nuke:shortname: smithi131 2024-01-13T08:25:59.834 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-13T08:25:59.838 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi099.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515725', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-13 08:00:42.270691', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGf7JtgD1eHBuWcCD/jYn9Ug/25uiQOIu51R/HNGaetszbYiIviq+iiBCD8rF9ANJ0wOhRLXdavAViwRt2KnYsw='} 2024-01-13T08:25:59.861 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi131.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-12_21:52:25-rados-wip-yuri2-testing-2024-01-12-1128-pacific-distro-default-smithi/7515725', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-13 08:00:42.273198', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFStCvrqjeTn1EnjfVBkPOdg++w7f7vX/2Ko2F4wPq9kDMHOP3I1aBrUEPOcyvXM2RoVC9l2fc2NItarP+zIN/o='} 2024-01-13T08:25:59.889 INFO:teuthology.orchestra.console.smithi099:Power off 2024-01-13T08:25:59.889 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-13T08:25:59.912 DEBUG:teuthology.orchestra.console.smithi099:power off output: Chassis Power Control: Down/Off 2024-01-13T08:25:59.912 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-13T08:25:59.928 INFO:teuthology.orchestra.console.smithi131:Power off 2024-01-13T08:25:59.928 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-13T08:25:59.937 DEBUG:teuthology.orchestra.console.smithi099:check power output: Chassis Power is on 2024-01-13T08:25:59.950 DEBUG:teuthology.orchestra.console.smithi131:power off output: Chassis Power Control: Down/Off 2024-01-13T08:25:59.951 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-13T08:25:59.975 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is on 2024-01-13T08:26:03.938 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-13T08:26:03.977 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-13T08:26:04.048 DEBUG:teuthology.orchestra.console.smithi099:check power output: Chassis Power is on 2024-01-13T08:26:04.086 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is on 2024-01-13T08:26:08.052 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-13T08:26:08.088 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-13T08:26:08.161 DEBUG:teuthology.orchestra.console.smithi099:check power output: Chassis Power is off 2024-01-13T08:26:08.197 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is off 2024-01-13T08:26:08.263 INFO:teuthology.orchestra.console.smithi099:Power off completed 2024-01-13T08:26:08.298 INFO:teuthology.orchestra.console.smithi131:Power off completed 2024-01-13T08:26:08.403 INFO:teuthology.run:Summary data: description: rados/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} duration: 1184.3604936599731 failure_reason: reached maximum tries (121) after waiting for 120 seconds owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=8b07fd114bf845febed910225c1b776c status: fail success: false 2024-01-13T08:26:08.404 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-13T08:26:08.483 INFO:teuthology.run:FAIL